TDIing out loud, ok SDIing as well

Ramblings on the paradigm-shift that is TDI.

Saturday, February 22, 2025

Couple of handy functions to share

First, I wanted to scan a folder and all sub-folders and return a JS object that reflected files found. The reason I often use JS objects to carry data is that they work beautifully and simply with the Javascript for-loop.

Here's what I got working:

// Recursive function to spin through a folder (java.io.File or String) 

// and sub-folders and return a JS fileListObject. Each property of the object 

// is a sub-object with its property name being the path to the file, 

// ensuring uniqueness for property names. Each object has the following three properties:

//

//  "\\FilesToFix\\Clip 08.dv": {

//    "folder": "FilesToFix",

//    "name": "Clip 08.dv",

//    "date": "20021224 18:34:18.953+0100"

//  }

//

// JS Objects are handy carriers of objects, and can be any primitive 

// Ecmascript 3 type or a Java object, like an Entry, Attribute or java.util.HashMap.

//

function getFiles(folder, fileListObjectArg) {

// Initialize the object if not passed in

var fileListObject = fileListObjectArg || {};

// If the folder name pass passed in, convert it to a File object

if (typeof(folder == "string")) { 

var useFolder = new java.io.File(folder);

if (!useFolder) throw "Cannot open folder: " + folder;

folder = useFolder

}

// Get the folder name and list of files in it

var folderName = folder.getName();

var fileList = folder.listFiles();


// If any files found, run through the returned list of java.io.File objects

if (fileList) {

for (file in fileList) {

var fileName = file.getName();

// If it's a directory then call this function recursively, 

// passing in the current state of the fileListObject

if (file.isDirectory()) {

getFiles(file, fileListObject);

} else

if (fileName.trim().toLowerCase().endsWith(".dv")) {

var dateString = system.formatDate(new Date(),

                                                                           "YYYYMMDD HH:mm:ss.SSSZ");

fileListObject[file.getPath()] = {

date: dateString,

name: fileName,

folder: folderName

}

}

}

}


return fileListObject

}

I am able to use a JS Object as a data carrier (think Entry holding Attributes, just lighter) as long as each property that I add has a unique name, and using the filepath as the property's name ensures that.

To easily check the results of this function, I translated a nice Java example that I found on Stacktrace for beautifying JSON, and vióla:

// Pass in JSON string or JS Object, it is returned as beautified JSON

//

function prettyJSON(jsonString) {

// If object is passed, convert it to a jsonString

if (typeof(jsonString) == "object") {

jsonString = toJson(jsonString)

}

// Import Java classes

var ObjectMapper = new Packages.com.fasterxml.jackson.databind.ObjectMapper;

// Now get an Object Mapper

    var objectMapper = new ObjectMapper();

    // Need to use ObjectMapper to create our JS Object here

    var jsonObject = objectMapper.readValue(jsonString, java.lang.Object);


    // Return the beautified code

    return objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(jsonObject);

}

The reason I am doing all this is that I have gigabytes of old pictures and movies that I (thankfully) sorted into folders and with date info captured in the folder names.

Now, I would like to run through my gigabytes of files and set the Date Created metadata value based on the date encoded in the enclosing folder's name. Then I can more easily import them into my main photos library and they will get sorted into the timeline correctly.

My testing code looks like this:

var filesListObj = getFiles("\\FilesToFix", {});

task.logmsg(prettyJSON(filesObj));


// Run through the filesListObj

for (var fileName in filesListObj) {

var file = filesListObj[fileName];

task.logmsg("--> " + fileName

    + "  date: " + file.date

    + "  folder: " + file.folder

    + "  name: " + file.name)

I'll stop here for now and starting thinking about how to set the Date Created metadata for each file I process. Maybe the TDI LabJam GPT has suggestions.

Until next time.

Thursday, November 28, 2024

Back again

After a hiatus these past few years, I am again getting to focus on TDI (SDI, Metamerge Integrator - call it what you want, a beloved child has many names, from the Norwegian proverb). Anyway, I have published a TDILabJam GPT to the public on ChatGPT where I have uploaded the systems docs, some tutorials and the javadocs. 

Be aware that although it has provided value these past few weeks, it is still just an LLM. Or as Apple researched showed, an advanced pattern matcher. So be prepared to read code, even if you don't have to write so much. I hope it can help you. It does not have all the docs for Adapter development, as the GPT seemed to have trouble remembering stuff with additonal docs. I think the context got too big. So I thought to save this for a possible separate GPT. Note that you can always just upload a PDF or two and then Chat about it in most of the major products, like Claude, Gemini, Llama, etc.

In the meantime, I hope the current asset can provide value. Here is the link to the GPT: https://chatgpt.com/g/g-yhIDQaJjh-tdi-labjam

Note also that there is a new public TDI community site and forum (again, called TDILabJam) on Reddit: https://www.reddit.com/r/TDILabJam/

In addition, a small group of TDIers has begun to assemble online on Tuesdays at 16:00 CET (roughly twice a month for now). The idea is to exchange ideas, look at challenges together and (really) to give me a chance to teach again :)

Also, if there are specific things that avid TDIers out there want to discuss, drop your suggestions into the #general sub-reddit. This the only one in our Reddit community at present. Or as comments to any of my posts or YouTube videos.

Also, don't forget that there is http://www.tdi-users.org where there is still a wealth of tutorials, How To's, Q&A and more.

Ah, it's good to be back!

Monday, October 4, 2021

As Eisenhower said, Plans are Useless

...as soon as the first shot is fired. However, the art of planning is invaluable. In other words, plan for change and be ready to change your plans. Repeatedly. This be the foundation of an agile approach to solution development.

I agree with Ken Schwaber in that many agile methodologies do not feel very agile. Some major contenders are thinly veiled command-and-control frameworks in which you entrust your development effort. This kind of regimented approach is seen as a less risky flavour of agile for newcomers to the game. And it feels familiar to those who followed the phase-gate approach to development which was dominant in the last millenia.

Teams should spend focus and energy on meeting Sprint goals, as opposed to rigid adherence to, and maintenance of the process. And sure it’s easy for me to say. Over the past twenty years I’ve had the pleasure of working in dozens of teams, and most of them were comfortably small. Single-pizza sized. However, I have also clocked hours in larger projects and participated in dutiful loyalty to formally defined tasks and metrics gathering, all the while as velocity floundered. With few degrees of freedom to analyze and optimize the process itself, we lacked the tools to improve - at least during my residency.


In my experience, agility that matters requires embracing change. Enabling and amplifying feedback, and then exploiting this to drive velocity and quality. Which is why I am such a fan of Scrum. As Jeff Sutherland says in his seminal book, Scrum is ‘akin to evolutionary, adaptive, and self-correcting systems’. The founders of Scrum live by this ethos of relentless improvement and the Guide itself weighs in at 13 pages! It represents the essence of agility.

What I am trying to impress on my fellow practitioners in the elusive art of agile solutioning is to keep the Agile Manifesto close to your hearts, regardless of the methodology your dev org follows. Strive for the three pillars of Scrum: transparency, inspection and adaption - of the product and the practice. It’s too easy to lose sight of the forest by focusing too much on individual shrubberies




Sunday, December 13, 2020

Agile is as Agile does...

...to paraphrase Forest Gump. TDI (aka SDI) is an extraordinarily agile platform for exploration, prototyping and crafting enterprise-strength solutions for data movement and services. However, using it or any other tool does not an agile developer make. Agile development is all about mindset and culture rather than processes and tools. In this age of industrialized and weaponized Agile (the noun vs. the adjective, as opined by thought leaders like Ken Schwaber and Dave Snowden) that perspective can be lost.

But of course, seeing the world through TDI-colored glasses mean that my view is indeed a bit colored...

TDI is an excellent platform for agile development work, also in teams. I've worked in several projects where multiple developers worked on the same TDI projects. However, sharing the source code (ALs, Components, Scripts, etc.) using something like git requires a little forethought and setup. The first step is good organization of both source and deployment assets.

As I've noted before, I organize projects in a particular way that has served me well for years:
  • A Project Folder (the one that Eclipse gives you in your workspace and that shows up in the CE). In general, apart from running git from this folder, I tend to leave everything being managed by Eclipse alone.
  • And a Solution Folder (with the same name as the Project itself and found in the Solution Directory). 
  • Furthermore, all filepaths in the solution are relative and reference files in the Solution Folder, or its sub-directories. Filepaths also have only forward slashes instead of backslashes.
The advantage of this last item is that solutions run equally well on Linux and Windows, and that you can deploy the solution to a new server by first zipping down the Solution Folder and then unzipping it into the Solution Directory of the target TDI server. Of course, along with any CustomJars files and folders your solution depends on.

As you may have gathered from my description above, there needs to be two git repositories: one for the Project Folder and  one for the Solution Folder. The Project Folder is the source while the Solution Folder holds the deployment assets, including properties and other support files and folders. Which also means all Property Store filepaths should be relative (and with forward slashes). As noted above, you may have custom jars for your solution. Simply drop them all into a folder ,  e.g. CustomJars, in the Solution Directory. You let TDI know about this folder via the com.ibm.di.userjars property in solution.properties which is found in your Solution Directory. Be sure and make this a relative path too.

This is the first step. Then comes the tricky bit.

Since TDI source files (*.assemblyline, *.connector. etc.) are all XML encoded, merging changes when two or more people edit the same source asset can be a challenge. For one thing, there are plenty of XML tags in these files that are only for internal use, like <ModTime>. So you end up having go through the files, comparing known tags and script snippets. Not ideal.

Which is why we kept to a strict git regime: push to the trunk (i.e. master, with no branches) using pull requests for peer review, and agreement on which files we each were working on. Not ideal, but until we can get a VSCode extension that handles this for us, it's the best you can do. Of course, editors like VSCode make it easier to compare changes and accept/reject as desired.

Once source management (and sharing) is organized, it then comes down to organizing work. Here is where Scrum, the brainchild of Ken Schwaber and Jeff Sutherland, both Agile Manifesto signatories, comes in.

Their original vision of Scrum is as a flexible, adjust-as-you-need methodology that really dictates only a few things:

  1. Cross-functional, empowered teams
  2. Short work cycles with clearly defined outputs
  3. Open and continuous communication
The first point is to limit wait times due to expertise or authority lacking in the team. The second item is so that measurable results are continually produced. And finally, the last point is to make sure that a) any blockers can be quickly swarmed on and dealt with, and b) everyone in the team knows what is happening, current priorities and why.

Now you might be saying, hey wait, traditionally practiced Scrum also defines distinct rolls like product owner, dev team and Scrum Master. These roles may be covered by team members, although for small teams they may necessarily be resources outside the team. If so then it is important to open lines of communication with the people performing these duties - and one that provides for informal comms if possible. 

Most of the TDI work I've done over the past couple of decades has been with one to two other individuals. So to augment our team, we enlisted someone with both an understanding of the requirements, and the ability to prioritize these. Typically an architect or project manager, who then becomes our Product Owner. As to the Scrum Master, I've provided agile coaching while we all shared the admin work.

And finally, we build a relationship with the person or people responsible for operating the solution once we deliver it. This gives us DevOps insights into how we make the final solution easier to (re)install (e.g. Disaster Recover), manage (stop & restart) and monitor (logs, audits and other operational output).

Tuesday, December 1, 2020

How to turn security events from QRadar into self-healing actions on infrastructure

Here for your perusal is a self-contained solution that provides a service AL that listens for incoming requests coming from QRadar Event Rules. You write your own ALs to perform actions based on various events, using data shared by QRadar in the event action request JSON. Note that this is part of the Guardium bundle, leveraging it to automatically perform reconfiguration, like locking suspicious user credentials, or closing off access to a resource pending further analysis.

https://github.ibm.com/eddie-hartman/QRTrigger

Enjoy!

Thursday, January 23, 2020

SDI, TDI, what's in a name

When IBM acquired the integration powertool back in 2002 it was called 'IBM Data Integrator'. Then after a bit of political squabbling, it was moved to the Tivoli brand and renamed 'Tivoli Directory Integrator'. Now, after its move to the Security brand a few years back, the name became 'Security Directory Integrator'.

I just think of it as 'Integrator', although my fingers still type 'TDI'. Old habits. And it is still being used for a wide variety of scenarios from cloud application integration to infrastructure rewiring.

So don't let the name fool you.

Monday, June 17, 2019

How To Ask Smart Questions

Although this article is particularly about asking hackers there are some universally relevant pearls here that are ring particularly true today:

http://catb.org/~esr/faqs/smart-questions.html

It's easiest to go for the quick fix when it comes to information gathering. It's apparent in the alarming number of people who's primary news source is social media, and it's highlighted nicely in this article about how Google Maps is making us dumber:

https://www.vocativ.com/418342/google-maps-gps-navigation/index.html

In particular, it's about exercising the gray cells to uncover the desired knowledge. This list of steps from the first article above is by no means exhaustive:

--- excerpt

Before You Ask

Before asking a technical question by e-mail, or in a newsgroup, or on a website chat board, do the following:
  1. Try to find an answer by searching the archives of the forum or mailing list you plan to post to.
  2. Try to find an answer by searching the Web.
  3. Try to find an answer by reading the manual.
  4. Try to find an answer by reading a FAQ.
  5. Try to find an answer by inspection or experimentation.
  6. Try to find an answer by asking a skilled friend.
  7. If you're a programmer, try to find an answer by reading the source code.
---

Of course, sometimes speed is critical. But a little forethought and investigation will help make your question easier and faster to answer.

And since I see the world through TDI-colored specs, I going to add that


  • You include 'java' to any searches involving TDI research and troubleshooting (step 2)
  • The AL Debugger is a great tool for step 5
  • The TDI user forum is where many of your skilled friends can be found* :)


*Link here: www.tdi-users.org