TDIing out loud, ok SDIing as well

Ramblings on the paradigm-shift that is TDI.

Saturday, February 6, 2010

What Is The Solution Directory?

The Solution Directory is the personal working folder for a TDI Server. Each TDI Server running on the same machine must have it's own Solution Directory. The reason for this is that a TDI Server gets it's configuration settings, including settings for the JVM, from a file in the Solution Directory named solution.properties. After reading the solution properties file then the Server reads the global.properties file in the etc sub-folder of the TDI installation directory. Settings in solution.properties override those in global.properties.

If you open solution.properties in a text editor you can find two important properties that are personal to any running TDI Server:

  • The port that the Server listens to for RMI API calls: api.remote.naming.port
  • The port that the Server listens to web calls: web.server.port (dashboard, FDS, CURI and REST APIs)

This is also where you provide the paths to keystores and truststores. By default, those for the TDI Server itself are named 'testserver' and found in the Solution Directory. Those used for SSL connections are by default under the serverapi sub-folder and are called 'testadmin'. The password for 'testserver' is 'server, and for 'testadmin' is 'administrator'. To import your own certificates use keytool (found under the jvm/jre/bin sub-folder of your TDI installation directory) or the Key Manager button in the TDI CE (Configuration Editor).

If you remove all properties in solution.properties that you are not explicitly setting, then a TDI version upgrade could affect settings for your Server.

Leveraging the Solution Properties folder to make your solutions more portable

The Solution Directory is also the "current path" whenever you run TDI, either the CE or the Server. As a result, if you use relative filepaths then your solution is easily moved between TDI Servers. For example, edit the Editor tab of your Resources > Properties > <Project PropertyStore> and set the Collection Path/URL to a filepath that is relative to your Solution Directory - for example: MyProject/MyProject.properties. Also, always use forward-slashes for paths, that way your solution runs on Windows and *nix.

Once your properties are stored relative, right-click on the topmost folder of your Project in the CE Navigator View and select Properties. Choose Tivoli Directory Integrator properties at the bottom and set the Linked File parameter to a relative path that points to a config file under this same folder, like MyProjext/MyProject.xml.

Make sure you also put any other support files in this folder so you can easily move the solution by simply compressing this folder and then uncompressing it in the Solution Directory of the target TDI installation.

The ReadMe.txt file you make in your project directory could look something like this:

To install this solution:

1) Unzip ADSync.zip to your Solution Directory;
2) Edit ADSync.properties to configure connections
3) Run the solution with this commandline:

ibmdisrv -c ADSync/ADSync.xml -r ADSync

Viol√°, you have something that you can hand to the operations team for deployment, pass on to a colleague, and even publish it online (and get yourself a Metamerge pen by sending me the link).

Pointing to a specific Solution Directory

Stephen Swann pointed out to me that I need to mention the ibmdisrv and ibmditk commandline argument for specifying the Solution Directory to use: the -s argument. In additon, in those cases where TDI has been installed via automated scripts and the CE is not going to be started in order to prepare the Solution Directory, there is also the -g argument to instruct the TDI Server to do this job.

    ibmdisrv -s /MyOtherSolDir -g

The above command line will instruct the TDI Server to prepare the MyOtherSolDir folder as a Solution Directory.

Your Eclipse workspace

So the Solution Directory is a tool for making solutions portable, and it is independent of where you put your TDI 7 workspace. The workspace is where the Eclipse part of TDI organizes your source files, and the directory structure there mirrors what you see in the CE Navigator. It is not recommended that you work directly with source files, apart from tying them to a Source Management tool.

What if..?

But what if you chose the program directory or "current location" when you installed, or want to move the Solution Directory for some other reason? No problem. Go to the TDI installation folder and edit the batch-file (script) bin/defaultSolDir and change the path set there. This single line sets up the TDI_SOLDIR environment variable used further down in the launcher script. So if you want to move your Solution Directory, simply change this assignment. Note that if you craft your own scripts/batch-files, also set TDI_SOLDER at the top and then refer to this variable to invoke TDI launch scripts. This makes your own scripts easier to re-use.

Changing TDI_SOLDER is only half the job. You also have to instruct the CE where the new Solution Directory is when it automatically starts up your local TDI Server, called 'Default'. You do this by right-clicking on 'Default' in the Server view and selecting Properties and making the change here too.

Final note about the Servers view

The view is not called Servers for nothing. TDI lets you define new Servers, allowing you to start multiple Servers on your PC, or connect to those running on other platforms. You can point a Project at any Server by editing its settings (right-click > Properties). This lets you test and debug the ALs there. Each Server has its own Solution Directory settings, so using relative paths means you can build your ALs on your laptop and then run them elsewhere. You will also want to use forward slashes for filepaths, since backslash only works on Windows and TDI runs a lot of other places as well.

And while I'm on the soapbox here are some additional tips: Define Logging for where your task.logmsg() messages are written; Code Error Hooks and log enough info for manual intervention; Write status messages to the command line by using main.logmsg(); And set up Connector Auto-Reconnect (Connection Errors).

21 comments:

Anonymous said...

Interesting article you got here. I'd like to read something more about that topic. Thank you for giving this information.

Eddie Hartman said...

Anything in particular? I'm taking requests :)

BliZzArD182 said...

I would love to have more information on passwords sync! Specially MSAD passwords.

Use the force!

Eddie Hartman said...

What would you like to know? TDI Identity Edition includes password change interceptor plugins for (among other systems) AD. This plugin grabs password changes and securely sends them to an AssemblyLine you prepare for writing the credential updates to your targets.

The transport for sending the password change message is either a queue (MQe, MQ) that your 'SyncPassword' AL reads; Or by having the plugin write password change entries to a special container/branch you set up in an LDAP directory where you pick them up using a Changelog Connector. Easy as spittin' :)

More here:
http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/topic/com.ibm.IBMDI.doc_7.0/pluginsguide.htm

Anonymous said...

Good brief and this enter helped me alot in my college assignement. Gratefulness you seeking your information.

Stephen Swann said...

Eddie,
It seems to me that many enterprises will have a need to deploy TDI using automated build scripts and may have a need to create (and populate) a solution directory from the command line rather than firing up the CE. For those people in that situation, the "ibmdisrv -s {soldir} -g" is your friend.

Eddie Hartman said...

@Stephen Swann
Absolutely. Much of what you make in TDI will be left behind for others, with perhaps less TDI-savvy, to run/automate/manage. So you need all the friends you can get :)

Dan Rolander said...

Hi Eddie. When multiple solution directories are required, what is the best approach? I understand how to set the solution directory at startup time, but how should they be organized? Is it acceptable to have a parent directory as the default solution directory, and then subdirectories that are setup as their own solution directories? Also, it seems that in TDI 7 there is a separate Derby Database created for each solution directory, is that correct? What are the ramifications of having multiple Derby databases running on a single server? I assume they would all need unique ports, is that correct? Seems like it could be complicated to shutdown and startup all dbs for server patching, etc.

Eddie Hartman said...

@Dan Rolander
You are right that it can get messy. And there should be a way to have multiple TDI Servers share the same Solution Directory so that you can use relative paths to stuff like Properties and other support files. However, I just discovered that there is a bug - which will get fixed :)

But in the meantime, your idea of having a parent folder which contains your support files (e.g. props) - or perhaps a sub-folder that is shared with these files - and then a separate sub-directory for each TDI Server's Solution Directory, so they each get their own solution.properties with port addresses. Then your solution relative paths can all be like this:

../MyProject/MyProject.properties

Etc. And you can put a single System Store here as well which all the TDI Servers share. Just make sure you run Derby in Networked Mode. This is default now for TDI 7, but you will need to configure TDI 6 according to the docs. I would definitely not run multiple Derby instances if you can avoid it. And with networked mode, then the first TDI Server that needs the SysStore will start it - the others will just connect.

Or if you are using TDI 7.1 or later then you could opt to use Solid DB instead. This is a memory DB which is considerably faster than Derby - and more robust. It can also be easily setup for HA. Or you could opt to use an RDBMS that you have running in your infrastructure already: DB2, Informix, Oracle, SQL Server, etc. The SysStore can be pointed at any JDBC compliant db server.

Hope this helps!

Brent Crosling said...

Hi Eddie

In a TDI 7.1.x Linux environment, would you see any complications with having one ibmdisrv process running (with, -d &) and having only one TDI solution directory, and then running all required TDI configs from it's configs folder?
My view of this would be using tdisrvctl to start/stop any of multiple TDI configs involved; though I can see the ibmdi.log being overloaded with detail due to one ibmdisrv process running. Also, would this then keep the TDI message queue/store work to a minimum?
I know the ITIM Adapter starts it's service with ibmdisrv (via ITIMAd) and not using tdisrvctl, so I would think that if the ITIM Adapter is to be run from the scenario I mentioned before, then I would create a new script (using tdisrvctl) to start the ITIM_RMI.xml TDI config.
I believe the scenario I mention would also lead to the tdisrvctl command showing a complete list of loaded configs and their assembly lines.

In summary, I'm essentially trying to run a single server TDI environment (and single TDI instance) with many configs being able to run effectively.

Interested to hear your thoughts.
Thank you.

Brent Crosling said...

Hi Eddie

In a TDI 7.1.x Linux environment, would you see any complications with having one ibmdisrv process running (with, -d &) and having only one TDI solution directory, and then running all required TDI configs from it's configs folder?
My view of this would be using tdisrvctl to start/stop any of multiple TDI configs involved; though I can see the ibmdi.log being overloaded with detail due to one ibmdisrv process running. Also, would this then keep the TDI message queue/store work to a minimum?
I know the ITIM Adapter starts it's service with ibmdisrv (via ITIMAd) and not using tdisrvctl, so I would think that if the ITIM Adapter is to be run from the scenario I mentioned before, then I would create a new script (using tdisrvctl) to start the ITIM_RMI.xml TDI config.
I believe the scenario I mention would also lead to the tdisrvctl command showing a complete list of loaded configs and their assembly lines.

In summary, I'm essentially trying to run a single server TDI environment (and single TDI instance) with many configs being able to run effectively.

Interested to hear your thoughts.
Thank you.

Eddie Hartman said...

@Brent
I would not combine a TDI instance being used by TIM with other TDI work. TIM has its own Dispatcher engine running inside the TDI Server, and I don't know enough about this guy to speak to safe co-existence.

Use a single TDI instance to power several other TDI solutions should be no problem. You can adjust the server log settings in the /etc/log4j.properties file.

Unknown said...

Hey man, I'm tryin to migrate AL from v6.1.1 to V7.1.1 but all my Als are creating the TDISysStore under V7.1.1 path and not in solution dir, i have checked solution and global porps under al folder, they have this:
com.ibm.di.store.database=TDISysStore

But not sure why they keep trying to build TDISysStore under /opt/IBM/TDI/V7.1.1

Thanks.

Unknown said...

Hey man, I'm tryin to migrate AL from v6.1.1 to V7.1.1 but all my Als are creating the TDISysStore under V7.1.1 path and not in solution dir, i have checked solution and global porps under al folder, they have this:
com.ibm.di.store.database=TDISysStore

But not sure why they keep trying to build TDISysStore under /opt/IBM/TDI/V7.1.1

Thanks.

Eddie Hartman said...

@Unknown

In TDI 7.1.1 you can right-click on the Default server and choose 'Edit system store settings'. In the resulting editor you will notice a tiny arrowhead to the right of the editor title: Server System Store. Click on this to select one of the out-of-the-box support SysStore settings. Yes, this will update your solution.properties, which you've said you have done already. However, it will allow you to compare what you set with what the system expects to see.

Hope this helps!

Unknown said...

Hey,

Sadly i have no editor installed on my laptop, i work against TDI running on AIX.

What I did is:

com.ibm.di.store.database=$soldir/TDISysStore

and that worked, the issue now is, derby.log is being created still under /opt/IBM/TDI/V7.1.1, SO I'm assuming this ne version is skipping some of the properties I have already on v6. Will keep working.

Eddie Hartman said...

@Unknown

Depending on which TDI version you have, you can define the TDI Server running on AIX in the Servers view of the CE running on your laptop. Then you can access all the server functions.

Unknown said...

Hey Eddie, It's Ivan again, These days have been TDI learning days, managed to fix everything but one issue left. You may have the solution.

On old logs I used to see this:

2014-07-03 05:35:27,393 DEBUG [AssemblyLine.AssemblyLines/SSOQ2TAM.1962308854] - [TAM_LDAP_Lookup_1] CTGDIS064I Loading Attribute Map.


So I can see logger is recognized as AssemblyLine.AssemblyLines/SSOQ2TAM.1962308854


But since migration, I see this way:

2014-07-04 01:27:42,439 DEBUG [com.ibm.di.log.FileRollerAppender.4c2cb50c-9550-41cc-94d1-ffd9d1abc144] - CTGDIS082I Load: DisplayData2.

So logger is now recognized as com.ibm.di.log.FileRollerAppender.4c2cb50c-9550-41cc-94d1-ffd9d1abc144

which is uglier, nastier, and it's pissing me off to see my logs that way.

Do you have any idea where that is coming from?
I have to add that the format of the first line is also shown in ibmdi.log after migration:

2014-07-04 02:14:42,942 INFO [AssemblyLine.AssemblyLines/SSOQ2TAM.1] - [Peek_MQ] Snoozing for 60000 ms..


But it is not shown on my custom logs:

2014-07-04 02:14:42,942 INFO [com.ibm.di.log.FileRollerAppender.4c2cb50c-9550-41cc-94d1-ffd9d1abc144] - [Peek_MQ] Snoozing for 60000 ms..


As you can see, it's the same msg but somehow it does not recognize its name.Since all files are the same, I'm assuming something is not being loaded good or TDIv7 does it different.

Eddie Hartman said...

@Unknow (aka Ivan)

Yes, Ivan, logging is different in the newer version. Have you tried playing with the pattern for the logger? I found this link: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html

See if this helps you get the output you want. If not, then let's move our conversation to the TDI forum, since there will be more eyes on your question and more nimble brains to reflect on an answer. The link can be found here: http://www.tdi-users.org

Ashok said...

Hi Eddie,


I have a requirement to run 2 AssemblyLines with 2 different changeLog connectors(SUn LDAP and ISDS) on the same TDI Machine. Both the change log no's are in different sequence. However i can run only one AL correctly. The issue here is, my Second AL is also picking up the previous AL's Change log no and trying to pull the information accordingly.

Can you please guide me to overcome this conflict and run both the AL's individually using it's own changelog no's.

Thanks
Ashok

Eddie Hartman said...

@Ashok I don't see how the two ALs can interfere with each other - unless you are using the same Iterator State Key for both Change Detection Connectors. Note that this parameter value must be unique for each CDC.

Apart from that, each AL is a separate thread in the JVM and it sounds like you are using two different types of CDC as well. Maybe you need to share more details on your setup. And I would post questions like this to the TDI forum instead of YouTube comments. Here is the link:

https://groups.google.com/forum/#!forum/ibm.software.network.directory-integrator