tag:blogger.com,1999:blog-44768638001584001702024-03-05T05:08:15.755+01:00TDIing out loud, ok SDIing as wellRamblings on the paradigm-shift that is TDI.Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.comBlogger77125tag:blogger.com,1999:blog-4476863800158400170.post-86943717229077985902021-10-04T20:21:00.006+02:002021-10-15T20:11:17.165+02:00As Eisenhower said, Plans are Useless...as soon as the first shot is fired. However, the art of planning is invaluable. In other words, plan for change and be ready to change your plans. Repeatedly. This be the foundation of an agile approach to solution development.I agree with Ken Schwaber in that many agile methodologies do not feel very agile. Some major contenders are thinly veiled command-and-control frameworks in which you Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-71622614510795507882020-12-13T11:43:00.000+01:002020-12-13T11:43:03.611+01:00Agile is as Agile does......to paraphrase Forest Gump. TDI (aka SDI) is an extraordinarily agile platform for exploration, prototyping and crafting enterprise-strength solutions for data movement and services. However, using it or any other tool does not an agile developer make. Agile development is all about mindset and culture rather than processes and tools. In this age of industrialized and weaponized Agile (the nounEddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com5tag:blogger.com,1999:blog-4476863800158400170.post-89758261595756932812020-12-01T14:04:00.000+01:002020-12-01T14:04:49.536+01:00How to turn security events from QRadar into self-healing actions on infrastructureHere for your perusal is a self-contained solution that provides a service AL that listens for incoming requests coming from QRadar Event Rules. You write your own ALs to perform actions based on various events, using data shared by QRadar in the event action request JSON. Note that this is part of the Guardium bundle, leveraging it to automatically perform reconfiguration, like locking Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-67825581824240942292020-01-23T09:21:00.002+01:002020-01-23T09:23:05.233+01:00SDI, TDI, what's in a nameWhen IBM acquired the integration powertool back in 2002 it was called 'IBM Data Integrator'. Then after a bit of political squabbling, it was moved to the Tivoli brand and renamed 'Tivoli Directory Integrator'. Now, after its move to the Security brand a few years back, the name became 'Security Directory Integrator'.
I just think of it as 'Integrator', although my fingers still type 'TDI'. OldEddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-70716200869451349272019-06-17T14:37:00.005+02:002019-06-17T14:37:57.137+02:00How To Ask Smart QuestionsAlthough this article is particularly about asking hackers there are some universally relevant pearls here that are ring particularly true today:
http://catb.org/~esr/faqs/smart-questions.html
It's easiest to go for the quick fix when it comes to information gathering. It's apparent in the alarming number of people who's primary news source is social media, and it's highlighted nicely in Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-72081660160653790422019-06-04T14:22:00.002+02:002019-06-04T14:23:39.849+02:00SDI/TDI as a Windows ServiceNOTE: You must remember to copy the ibmdiservice.exe and ibmdiservice.props files from the <InstallDir>\win32_service folder to your Solution Directory (soldir) before you run the service installer.
It is a common mistake to run the service installer from the wrong folder - i.e. not your soldir. Often this is done from <InstallDir>\win32_service where the .Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-33124452892049181432018-09-25T16:08:00.002+02:002019-06-04T14:29:09.711+02:00Thinking about agilityMy favorite agile tool is TDI (and please forgive an old man for having trouble shifting to 'SDI'). Not only can I whip together integration service prototypes faster than anything I can do in Java (Spring), or Python or TurboPascal, but I can also build Ops features this way. Monitoring and remote control (pause, restart, failover/failback) for enabling auto-healing, or at least making the Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com1tag:blogger.com,1999:blog-4476863800158400170.post-57202464336973197622017-10-24T11:51:00.001+02:002020-04-24T13:42:05.986+02:00Collecting my ramblings on Connector Loops in one place
November 7, 2013 - Connector Loops and why you're gonna love them
September 30, 2013 - Why Loops Instead Of Lookup Mode
November 7, 2013 - Why Connector Loop
June 8, 2010 Loop a Connector today
Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-57852945371731049402017-09-01T01:14:00.000+02:002019-03-18T09:54:48.280+01:00TDI Getting Started and Ensuring Basic Understanding
The Basics
Install TDI (https://ibm.biz/BdjsLw) locally to your laptop. This lets you customize your TDI environment easier. If you can't get a licensed version, start with the free version (https://ibm.biz/BdjiMg). It's fully functionally, will give you time needed to get started solving with TDI, and the license file can be replaced when you get yours.
Do NOT install under Program Files on Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com7tag:blogger.com,1999:blog-4476863800158400170.post-37615586254163229382016-12-13T14:25:00.003+01:002016-12-13T14:25:49.627+01:00How to make the Search/Reconcile AL for an RMI Adapter against a REST or WS API
Writing RMI Adapter AssemblyLines can be tricky at times, especially when it comes to the Search AL - the one used for Reconcile operations. This is because the Dispatcher calls the Search AL in Cycle mode, which causes it to work differently than you might expect.
The usual behavior of an AL is that an Iterator in the Feed section delivers data - one Entry at a time - into the Data&Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com2tag:blogger.com,1999:blog-4476863800158400170.post-42629573814429794372016-12-12T12:47:00.000+01:002016-12-13T10:19:50.983+01:00Importing a client certificate in order to access an https URLWhen you go to make an HTTP request to a service that requires an SSL connection, you'll get an exception that your HTTP Client Connector cannot set up the secure link to the service. Unlike your browser, TDI (SDI) does not automatically import the client certificate. You have to do this yourself, and this post is about how to do this.
For this example, we'll try to access https://Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com8tag:blogger.com,1999:blog-4476863800158400170.post-50211327749180280102016-11-21T14:29:00.001+01:002016-11-21T14:29:08.697+01:00JSON and XML Tutorial - Part 6, Making a custom FormAs promised, this post will walk you through creating a custom Connection Tab Form for the Connector scripted in the previous post. Myself being in true form, I will begin with a promised explanation and then a bit of theory.
Internal name of a component
In my last post I promised to show you how to discover the internal name of a component. Otherwise, how can you know that the HTTP Client Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-61132223855661909472016-11-17T15:23:00.000+01:002016-11-18T15:26:58.408+01:00JSON and XML Tutorial - Part 5, Scripting a ConnectorContinuing from my last rambling post, we'll script an Iterator Connector that will return the same parsed entries as our Conditional Loop did. I'll begin with a little theory, which you will find duplicated other places like the How-To page of www.tdi-users.org and this venerable video tutorial on the subject.
To script a component in TDI (SDI) you start with the basic Script version of that Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com8tag:blogger.com,1999:blog-4476863800158400170.post-33755088872832509412016-11-16T15:36:00.002+01:002016-11-21T11:32:29.698+01:00JSON and XML Tutorial - Part 4, JSON example
Practice examples are helpful when mastering new techniques. In this example we'll make a REST request and then handle the return JSON payload. We'll be using the service available at the following URL:
http://www.ideal-status.nl/static/consumer_notification_advice.json
If you dial up this URL you'll get JSON that looks like this:
{"0021":{"issuer_id":"0021","issuer_bic":"RABONL2U","Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com15tag:blogger.com,1999:blog-4476863800158400170.post-33500870666781900822016-11-14T15:46:00.002+01:002020-04-12T13:14:06.628+02:00CouchDB/Cloudant ConnectorThose of you who want to learn more about the truly marvellous concept of MapReduce (thanks, Google!) or just want to play with the can't-believe-it's-free CouchDB, here is a connector I wrote a while back that supports both CouchDB and Cloudant. All you need to do is download a free copy of CouchDB (painless install and loads of fun), or sign up for your free Cloudant db in the cloud at Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com2tag:blogger.com,1999:blog-4476863800158400170.post-19102668666494000052016-02-22T12:13:00.000+01:002016-02-22T12:16:28.868+01:00JSON and XML Tutorial - Part 3, XML example
Continuing on from Part 1 of this tutorial, let's play a little with XML data using the HEntry techniques discussed.
Here is some example XML that we'll play with:
<teams>
<team name="Stark">
<sigil>Direwolf</sigil>
<member position="pitcher">Robb</member>
<member position="crowsnest">Bran</member>
</team>
<team nameEddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-85999865969707813632016-01-13T18:43:00.000+01:002016-01-13T18:43:02.805+01:00Externalizing an Attribute Map to a fileHit the More... button at the top of the AttMap - to the right of the Add and Delete buttons - and select Change inheritance. In the resulting dialog click External attribute map and select a file.
The external mapping feature of TDI uses text files that
contain one or more mapping rules. Each mapping rule looks like this: name=assignment.
Here is an example of a mapping Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-31121155932064970632015-04-10T15:55:00.000+02:002015-04-10T15:55:36.151+02:00What is the System Store?The System Store is a database that TDI uses to 'remember' stuff like:
the last change read, when you use one of the Change Detection Connectors (also called CDCs);
the snapshots of data used to compute changes when using the Delta Engine together with an Iterator Connector;
data read using the System Store Connector, which writes and reads full Entry objects with any number of Attributes;
Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-82982223890054099092015-03-02T19:52:00.002+01:002015-03-02T19:52:47.590+01:00Importing a client certificate from an https serviceI have cobbled together a simple AL that can be run from the commandline in order to import a client certificate from an HTTPS service. For example:
ibmdisrv -c configs/tdiingOL.xml -r importcert -0 https://supportcenter.checkpoint.com
The above call results in this output (including the standard TDI is startup messages):
CTGDKD024I Remote API successfully started on port:1099, bound to:'Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com1tag:blogger.com,1999:blog-4476863800158400170.post-82169331617528307612015-02-11T14:21:00.004+01:002015-02-11T14:21:56.323+01:00JSON and XML Tutorial - Part 2Well, this one is focused on JSON and I rolled it into a quick video.
https://www.youtube.com/watch?v=VHmf76FMI-U
Enjoy, and please comment if anything is unclear (or unsaid).Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-83169649850208906662015-02-05T13:55:00.000+01:002015-02-05T13:55:32.405+01:00Easily implementing a monitoring API to your TDI solutionAll you have to do is make AssemblyLines with the names of the operations you want to expose.
For example, I have a solution that catches incoming events over TCP and dispatches the payload data to one or more 'event triggered' AssemblyLines. Like all server mode based ALs, this one uses multiple threads to deal with client traffic (the AL Pool), and shared log output can get a little Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-61486372683683042482015-01-26T12:25:00.001+01:002015-01-26T12:25:21.507+01:00Passing arguments to your AL when launching it from the command-lineIn the ibmdisrv command-line used to launch your AssemblyLine, you can use one of the user-defineable options -0 through -9 to pass information into the AL.
For example, in order to pass the name of an input file to the AssemblyLine named 'ReadFile' you could use this command-line:
ibmdisrv -c MyConfig.xml -r ReadFile -0 c:/Input/HRExport.csv
Then in the Prolog - Before Initialize Hook Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-2794607662791565082014-11-24T13:27:00.000+01:002014-11-24T13:27:04.480+01:00Hardening an AssemblyLineThis is done with a little effort in three areas:
Error handling
Log handling
Auto-reconnect
For Error Handling you need to add code to the Prolog - On Error Hook for connection errors in components, and to the DataFlow - Default On Error (the catch-all Error Hook for all Connector modes). Here you will want to capture all error information to the log in order to help troubleshooting the issue.Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0tag:blogger.com,1999:blog-4476863800158400170.post-44768265265488635852014-08-04T11:49:00.004+02:002014-08-04T11:49:44.738+02:00Null Behavior
I've gotten this question (again) and decided to explain it here so that Google can find it.
Null Behavior allows you to deal with missing data without having to write Javascript. The Null Behavior feature lets you define what a 'null' attribute is and how it should be handled. By default the definition of 'null' is that the source attribute is missing or has no value (null value).Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com2tag:blogger.com,1999:blog-4476863800158400170.post-36184498144663690822014-07-12T11:25:00.000+02:002014-07-12T11:25:02.932+02:00mapReduce Revealed!Here is an excellent explanation of mapReduce that I've had to share (again and again). Enjoy!
http://www.slideshare.net/okurow/couchdb-mapreduce-13321353Eddie Hartmanhttp://www.blogger.com/profile/12264507239868838185noreply@blogger.com0