TDIing out loud, ok SDIing as well

Ramblings on the paradigm-shift that is TDI.

Monday, November 24, 2014

Hardening an AssemblyLine

This is done with a little effort in three areas:
  1. Error handling
  2. Log handling
  3. Auto-reconnect
For Error Handling you need to add code to the Prolog - On Error Hook for connection errors in components, and to the DataFlow - Default On Error (the catch-all Error Hook for all Connector modes). Here you will want to capture all error information to the log in order to help troubleshooting the issue. This can be done very simply by dumping he 'error' Entry to the log:

    task.dumpEntry(error);

The 'error' Entry holds a number of Attributes that indicate the error message and exception class, as well as where and when the error occurred. The 'exception' Attribute of the 'error' Entry is the Java exception itself, and you can print out the stacktrace using script like this:

   var sw;
   var pw;
   var ex = error.getObject('exception');

   if (ex != null) {
      try {
         sw = java.io.StringWriter();
         pw = java.io.PrintWriter(sw);
         ex.printStackTrace(pw);
         pw.close();
      } catch (ex2) {
         var sw = ex
      }


   task.logmsg("ERROR", sw.toString());

Note also that if you enable Error Hooks then execution will continue in the AssemblyLine as if nothing had occurred. So you will want to deal with the error yourself, for example by issuing a call like system.exitFlow() to stop the current cycle, or system.skipEntry() to pass control to the Feed Iterator in order to read the next entry. Or you may wish to re-throw the exception in some situations:

   throw error.getObject('exception');

Log Handling means making conscious decisions about how your ALs perform logging. Of course all log messages are added to the TDI Server's logfile (ibmdi.log). However, this can make it difficult to find and isolate messages, particularly if your solution has several AssemblyLines. Instead each AL should likely have its own log. My favorite LogAppender is the FileRollerAppender, which maintains a set of historical log files, rolling these by appending a number to the file extension each time the AL is run.

You may also want to provide a couple of log files for each AssemblyLine - one Appender set to log at INFO or DEBUG LEVEL, while another that only captures error messages - e.g. by setting the LogAppender level to WARN, which will include WARN, ERROR and FATAL.

I typically have my own scripted log() function that includes information about where in my solution the logging is taking place.

   function log(lvl, msg) {
      var toConsole = false;
      if (typeof lvl === 'undefined') { // if no arguments
         lvl = 'INFO';
      }
      if (typeof msg === 'undefined') { // if only one argument
         msg = lvl;
         lvl = 'INFO';
      }

      // In case the log level is not in uppercase
      lvl = lvl.trim().toUpperCase();

      // CONSOLE level = print to console
      if (lvl.startsWith('CON')) { // 'CONSOLE' log message
         toConsole = true;
         lvl = 'INFO';
      }
      // All error messages also go to the console
      if ('WARN'.equals(lvl) || 'ERROR'.equals(lvl) ||
          'FATAL'.equals(lvl)) {
         toConsole = true;
      }

      // Now to determine where log was called from.
      // First get the AL name
      var where = '[' + task.getShortName();

      // Afterwards comes the current component (if not an AL Hook)
      try {
         var compName = thisComponent.getName();
         where += '/' + compName;
      } catch (ex) {
         // was not in an AL component
      }

      // Now add the Hook name (if inside a Hook)
      try {
         var hookName = thisScriptObject.HookName.getValue();
         where += '/' + hookName;
      } catch (ex) {
         // was not in a Hook
      }

      // Log the message
      task.logmsg(lvl, where + '] ' + msg);

      // Print some messages to console as well
      if (toConsole) {
         java.lang.System.out.println(lvl + " - " + msg);
      }

   }

I have this function defined in a Resources > Scripts Script that is tagged to be implicitly included for all ALs.

Finally, Auto-Reconnect is a feature of Connectors and Function components found in the Connection Errors tab. Here you enable the reconnect feature typically only for when a connection is lost. The Number of Retries parameter is set to an adequate value, with the Delay Between Retries in seconds. In this way, if the connection goes down then the component can attempt to reconnect and then continue.

Note that the Reconnect rules must be set for this component. If you don't see built-in rules here then you will need to define your own, as described here.


Monday, August 4, 2014

Null Behavior


I've gotten this question (again) and decided to explain it here so that Google can find it.

Null Behavior allows you to deal with missing data without having to write Javascript. The Null Behavior feature lets you define what a 'null' attribute is and how it should be handled. By default the definition of 'null' is that the source attribute is missing or has no value (null value). The default handling is that an attribute with this name will be found in the target entry. So for an Input Map then the Work Entry will not have this attribute after the mapping is done. For an Output Map it will be the Conn Entry that does not have the attribute. Furthermore, if an attribute with this name was found in the target entry prior to the map, then it is deleted.

To illustrate this functionality, imagine you have an input map from a database connector with an attribute that gets its source value from a db column named 'TITLE', and that this column is nullable. In other words, not all rows need to have this column value. Alternatively, it could be an object repository (like an LDAP directory) and that the attribute in question is not found in all entries you are reading. During the mapping processes, SDI discovers that the source attribute (conn.TITLE) is not found. Null behavior will detect the 'null' and remove the attribute from the Work Entry.

Now imagine you are reading from a CSV file. In this case there are no missing attributes or null values, only empty string values for some attributes. So now you can change Null Behavior to define 'null' as being an empty string (plus all the other definitions over it in the Null Behavior dialog). Furthermore, you can define handling to be that you want a default value returned - for example N/A. ViolĂ , now all rows with an empty value for 'TITLE' will be returned with this attribute value set to 'N/A'.

Final note: Null Behavior can be defined at the map level using the More... button at the top of the map, or for a single attribute by right-clicking it and selecting Null Behavior.

Saturday, July 12, 2014

mapReduce Revealed!

Here is an excellent explanation of mapReduce that I've had to share (again and again). Enjoy!

http://www.slideshare.net/okurow/couchdb-mapreduce-13321353

Monday, May 26, 2014

Sometimes I dream in Javascript

Very nice article about big data, MapReduce and Javascript.

http://www.joelonsoftware.com/items/2006/08/01.html

And a rant I enjoyed greatly :)

http://steve-yegge.blogspot.no/2006/03/execution-in-kingdom-of-nouns.html

-E)

Thursday, November 7, 2013

Connector Loops and why you're gonna love them

I'm going to use an example where my AL needs to read User accounts and check if this user is a manager. If so, then it needs to search for people with the current user as manager. For each person found it will need to look up the HR information from another system.

Traditionally this would require in an Iterator in the Feed section to do the initial reading of User accounts. Then in the Data Flow would be an IF-Branch to filter out managers. Under this Branch comes a Connector in Lookup mode to find those people who report to this manager. For each person returned by the search there is another Lookup required to retrieve info from the HR system. Using the traditional approach will require a good deal of Hook scripting - both to deal with various lookp results (none found, and multiple found), as well as dealing with the final search in the HR system.

Connector Loops makes this much easier. Here is my AL to implement the logic described above:




Let's start with the innermost Connector Loop ('Get HR…') which is used instead of a stand-alone Connector in Lookup mode. As a result, I don't have to script (or even enable) the On No Match and On Multiple Found Hooks. Using a Connector Loop in Lookup mode still gives me option to code these Hooks if I choose, but even if I leave them disabled then the Loop will cycle once for each entry found. I may want to ensure that one and only one HR record was found for a managed person, but this is my decision based on the quality of the data I'm reading and the requirements of my solution.

The second point I wanted to make was that I set both Connector Loops (the 'FOR-EACH' components in the screenshot above) to Iterator mode instead of Lookup. Both modes do the same job - searching for data based on some filter - but they do it in different ways. One major difference is that Lookup mode caches the entries read, up to the number you set in the More... > Lookup Limit setting, while Iterator mode leaves the result set on the server and retrieves one at a time for each Get Next operation. Another important difference is that Lookup mode uses Link Criteria to control its search filter, whereas Iterator mode does its selection based on parameter settings. For example, an LDAP Connector uses it's Search Filter parameter, whereas a JDBC Connector will use the SQL Select parameter if it is set, and otherwise construct a 'SELECT *' statement based on the Table Name parameter. 

This last point leads me to another handy option for Connector Loops: the Connector Parameters tab.


This feature lets you use standard attribute mapping techniques to set the value of one or more parameters. In the above example I map to ldapSearchFilter to control which entries are selected by the Loop. 

Fortunately, the Search Filter is refreshed from the parameter settings whenever entries are selected, as they are at the top of this Loop for each AL cycle. However, most parameters are only updated with the values in the Connection tab once when the Connector initializes. If I needed to set a parameter that is not refreshed for selection, for example the File Path of a File Connector or connecting to a different database server, then I would need to configure the Loop to re-initialize as well as performing the selection.


The next thing I'd like to talk about is how you exit from Loops, as well as continuing to the next cycle of a Loop.

To leave a Loop you make a system.exitBranch("Loop") call in script, documented here: exitBranch. There are two variants of the exitBranch() call: one with no arguments that exits the current (innermost) Loop or Branch, and one with a String argument. If you use the predefined "Loop" argument then it exits the innermost Loop, skipping out of any Branches that might be closer in the AL tree. As the docs state, you can also exit a named Branch or Loop, as well as exiting the Data Flow section or even the entire AL.

On the other hand, if you want to continue to the next cycle of a Loop then you use the system.continueLoop() call, described here: continueLoop. Again you have two variants: one with no arguments that continues the innermost Loop, and one that lets you name the Loop to continue to.

Finally, note that the built-in looping performed by an AL - e.g. the Feed section looping iterated data into the Data Flow section - automatically empties out the Work Entry at the top of each cycle. A Connector Loop does not do this for you. If you want to handle this yourself then putting this snippet of code in the Before GetNext Hook of the Connector Loop (in Iterator mode) will do the trick:

work.removeAllAttributes();

You could use the same code in a Hook like Before Lookup for Lookup mode.

Furthermore, I often save off the work Entry prior to my Connector Loop, like this:

saveEntry = system.newEntry();
saveEntry.merge(work);

Then after the Loop completes, I can empty out the Work Entry once more and then merge back in the saveEntry Attributes.

And if you are still confused, please leave me comments and questions and I'll do my best to answer them.





Friday, November 1, 2013

My humble insights on TDI and source management

I've gotten this question a lot recently and so thought to share my thinking on the subject.

We here in the TDI team use RTC for shared projects. Due to technical difficulties getting TDI and RTC to run in the same Eclipse environment, I run them separately: TDI in my Windows image (Parallels) and RTC under OS X.

First step is to set up a workspace for RTC, which I call imaginatively enough 'workspace_RTC'. Here I copy in the project that I created locally - at least for projects where I do the initial development.

From RTC I import the project that was copied to the workspace_RTC directory. Only the AssemblyLines and Resources folders (and contents) are tagged as included in source management. Furthermore, under Resource > Properties you will only want to include the actual Property Stores designed for this solution.

Once this is done you can either delete the local copy of the project, or rename it. Then import the project in the workspace_RTC folder via the Existing Projects into Workspace option. Select the project folder and DO NOT select the option to Copy projects into workspace. That way the CE project is linked to the one in RTC.

Alongside the Project folder in workspace_RTC we usually have a second top-level folder where support files are kept and managed separately - i.e. solution deliverables. For example, property files, external scripts and stuff like externalized attribute maps. This folder also holds the 'compiled' Project Config xml. As you probably already know, every time you make a change and then Save or Run/Debug an AL in the CE, the config XML file in the Runtime-<project name> folder under your TDI workspace is updated. That's why we do NOT include the Runtime assets in the main RTC project. Otherwise all team members will constantly be (partially) updating this resource.

Instead, one person is given the task of refreshing his local project and then creating the compiled config XML file, which is then placed in the secondary folder. If this is my job, then I use the Project > Properties setting in my CE to Link to this file. That way, whenever I deliver changes, I also update the complete config XML.

After this, my day-to-day life consists of first retrieving any changes from RTC and then refreshing the project in my CE. After that I work on those items that are my responsibility and then deliver these to RTC. If others will be working on the same source files, you'll need to lock these resources until you're done with them.

Of course, if you are using one of the source management systems that can be plugged into a single Eclipse instance alongside TDI, life is a bit easier. Then you can use the Team option in the context menu for resources directly from the CE.

Wednesday, October 9, 2013

Dynamically changing Attribute Maps

One way to handle multiple Attribute Maps (either Input or Output) for a Connector during AL cycling is to do the mapping yourself via script. However, there are alternatives.

The easiest way to dynamically change a map is by defining multiple Attribute Maps under Resources in your Project and then swapping between these, for example in the Before Execute Hook of the Connector.

// first decide which map to use
// for example, "AttributeMaps/ComputerSystemMap"
// or just "ComputerSystemMap"
mapToUse = computeMapName(work)

if (mapToUse != null) {
   try {
      // true for Input Map, or false for Output
      thisConnector.useAttributeMap(mapToUse,true)
      return
   } catch (exc) {
      // report that the map is missing or invalid
   }
}


Another way is to use an externalized map file, which is a text file containing any number of lines. Each line represents a single mapping rule.


Mapping

Description

sn=
When the assignment is empty then simple mapping is used.
In the example rule shown in the left column, the attribute named 'sn' gets its value(s) from a similarly named attribute in the source Entry. For an Input Map this would be the 'sn' attribute of the conn Entry. For all other map types (Output Maps or Attribute Map components) the source will be the work Entry.
status='Updated'
Anything entered after the equal sign is considered the assignment of the mapping rule. It must be a snippet of Javascript, unless the Text with Substitution flag is used: {S}, described later in this post.

You can also reference variables or call scripted functions that are defined by the time the map rule is invoked.
In the example rule shown in the left column, the 'status' attribute will get the value specified by the Javascript snippet, resulting in the literal string value Updated


giveName=[
first = work.FirstName 

last = work.LastName 
return first + " " + last 
]
Using square brackets as shown allows the Javascript assignment to stretch over multiple lines.
You can also reference variables or call scripted functions that you have defined.
In the example mapping rule, a full name value is returned by concatenating the FirstName and LastName attributes in the Work Entry.

In addition, flags can be specified to control the behavior of a mapping rule. These flags must appear in curly braces immediately after the name of the attribute being mapped. Valid flags are: A, M and S.

The A and M flags correspond to the Add and Mod columns in the Output map of a Connector in Update mode, and control whether this attribute is mapped during Add and Modify operations. The S flag denotes that the assignment uses the TDI Text with Substitution feature, which allows for literal text values that can include one or more substitution tokens.


Mapping


Description

objectClass{A}=
The 'A' and 'M' flags are used to control when this attribute is enabled with regards to the Add and Modify operations of a Connector in Update mode. By default, attributes will be included for both Add and Modify operations unless only one is specified: A or M.
im The A flag shown in this example specifies that the 'objectClass' attribute should only be mapped for Add operations.


mail{S}={work.uid}@acme.com

The 'S' flag indicates that the following mapping rule uses Text with Substitution. As a result, any curly braces found will be replaced with the value of the substitution token specified in the braces.
For this example, the value of the uid attribute in the Work Entry is substituted for the token {work.uid} and then the literal string '@acme.com ' is appended to it. The resulting string is returned as the value for 'mail'.

Note that multiple mapping flags can combined by separating them with commas. For example: {M,S}.

Note also that in order for externalized map files to work correctly, you must apply Fixpack 3 to your TDI 7.1.1 installation.

By externalizing your map to a file, you can easily change the format of the file, or allow users of your solution to do so without having to fire up the CE. You can also swap which map file to use with the same code snippet show above - just make sure that the mapToUse variable contains the path to a valid file on disk.