Tag: API

Adding custom Calypso packages by adding a new jar file

After my very interesting and enlightening attendance at The Server Side Java Symposium last month, I’ve decided to start exploring new technologies and how better do this than by creating some Open Source tools and libraries.  Since I’ve been working almost exclusively with the Calypso API for the last decade, it’s somewhat logical that I would start there.  I decided that my first project is to integrate Calypso with GridGain as Dispatcher to distribute Risk Analysis execution on the Grid.   Calypso has its own Dispatcher implementation, but they themselves acknowledge it’s not ready for prime-time.   They do sell an adapter to plug into DataSynapse as well, but I figured an Open-Source alternative would be a worthwhile tool for Calypso implementors.  Besides, once I get it all up and running, I want to play with Scala as it seems to integrate very easily with GridGain.

So last week I launched Eclipse and created a new project.   As I began creating a new package, com.steepi, I paused.   In order to plug my package into Calypso, I would need to implement CustomGetPackages in the calypsox.tk.util package.  But if I do that, how is a Calypso implementor going to use it?   After all, they most certainly already have their own instance of CustomGetPackages!   Now granted, they certainly could make the modification to their class to add my packages.   Still.  What if I truly wanted to provide a library that would attach my packages simply by adding a jar to the CLASSPATH?   This problem merited further investigation…

After a few hours of research, I found a solution by locating an undocumented feature of Calypso.   During startup, the AppStarter class will at some point try to instantiate calypsox.apps.main.UserStartup.   If an instance of this class is found by InstantiateUtil, the method start() will be called via reflection.   Just the hook I needed!   I could place my custom code there to attach my custom packages.

Wait, though…

How would I go about doing so?   Calypso’s API does not provide a method to add packages to InstantiateUtil.   It’s all done within a static block when the class is loaded.   Thankfully, I’ve encountered limitations with the Calypso API before and the way to circumvent this problem is to use Java reflection to render methods and fields accessible.   Here, then, is the code that does exactly what I wanted!  If you compile the following code and add it to your CLASSPATH, you’ll be able to attach custom packages to existing Calypso implementation projects that have their own CustomGetPackages implementation. It’s a nifty way to provide a third-party library on top of Calypso, don’t you think?

1:  package calypsox.apps.main;
2:
3:  import java.lang.reflect.Field;
4:  import java.util.Collections;
5:  import java.util.List;
6:
7:  import com.calypso.tk.core.Log;
8:  import com.calypso.tk.util.InstantiateUtil;
9:
10: public class UserStartup  {
11:     public void start() {
12:         // In order to attach our packages, we operate on
13:         // InstantiateUtil class using reflection so as to
14:         // reach through its encapsulation.
15:         Class clazz = InstantiateUtil.class;
16:         Field field;
17:         List packages = null;
18:         try {
19:             // Retrieve the _packages field in InstantiateUtil
20:             field = clazz.getDeclaredField("_packages");
21:             // Since this is a private field, we need to set it
22:             // to accessible so we can access it
23:             field.setAccessible(true);
24:             // Get the value of the field via reflection.
25:             // This is the actual List object
26:             packages = (List)field.get(clazz);
27:             // Add Steepi packages so they are available when
28:             // instantiating through reflection
29:             packages.add(0,"com.steepi");
30:         }
31:         catch (Throwable t) {
32:             Log.error("Error", "Unable to locate InstantiateUtil._packages via reflection.", t);
33:             return;
34:         }
35:         // TODO: Apply same logic to update _invertPackages field as well
36:     }
37: }

As noted inline, you’ll want to do the same to update the inverted packages field. I’ve omitted the code for brevity.

This solution should work with any version of Calypso prior to 11.1.   Who knows whether or not they will eventually remove this logic since it is, after all, undocumented.   Still, for the time being, it should be easier to deploy my packages in existing Calypso implementations without needing any code change.   Sweet!

Note to Self: Use session token as argument across all your APIs.

If you’re gonna be running code on any kind of multithreaded environment, a service layer of any kind, whatever… In other words, unless you’re actually running on the GUI and you’re actually dealing with event-driven development… please please please add a session token as an argument to every method call in your API.

You probably will think: “Why? I don’t need it.” But if and when you do, it might be too late… because you might be live at several clients, and you may not be able to change the API at that point in time, or you have to be backward compatible, right? Well, if you ever plan on having two-phase commit, or horizontal access permissions, whatever… you better have some kind of session token that allows you to retrieve your “context.” If you don’t have that, you’ll hack it…. oooohhhh, you’ll hack it real good, because you won’t have a choice. But you know what? If you ever want a clean fix: you’re shit out of luck!

Think Outside the Box

I came upon an interesting problem late last week.

I’ve recently implemented an OTR Bond interface to Calypso so that, when new TIBCO messages alerts us of a new “On-the-run” Bond issue, we can insert the appropriate data into Calypso. Of course, if it’s a new Bond, there’s also a good chance that we don’t yet have the Bond definition in Calypso and, for that, we use the Bloomberg Connect API to retrieve the Bond and link it to the Calypso BondBenchmark… Sorry. That was just an overview but I can already see that most of y’all have your eyes all glazed over. Didn’t mean to bore you to death.

Anyway, I’m rather familiar with Calypso’s Bloomberg Connect API and how it works. Puh-lease… give me a hard one!!! After all, I pretty much wrote the whole thing single-handedly a couple of years ago. Let’s just say that a request file is generated, then FTP’ed over to Bloomberg. Then, it’s your responsibility to poll the remote directory for the equivalent response file. Once it is available, you download it and parse it locally with the data you requested. Pretty straightforward, right?

Well, when I implemented the Bloomberg interface, I thought of the “requested” scenario and never bothered (foolish me!) to think outside the box. Let’s say you request a Bond with CUSIP 123456, then I’d simply generate a request file called cy_123456.req (cy being short for Calypso. Y’see? To top it all off Bloomberg states that there’s a maximum number of characters in the file name. Some puny amount like 12 or 15, I think.) So anyway, cy_123456.req would get processed and FTP’ed to the remote directory. Then, I’d poll every once in a while for a file called cy_123456.out. When the file would be present, I’d FTP it back to my local directory and remove the file remotely, thus cleaning up after myself.

This ain’t rocket science, right? That should work just fine, no?

If you’ve got one instance of Bloomberg Connect running, it runs fine and dandy, yes… but here at Countrywide, we got a bunch of environments all running in parallel. We’ve got a couple of development boxes, UAT, and of course, the production environment. The configuration for these boxes are identical and the OTR Bond adapter task kicks off from 4 AM until 5:30 AM. So now you have these 5 or 6 processes all running on different boxes each listening to TIBCO. Let’s say at 4:05 TIBCO broadcasts: The 3 month OTR Bond has CUSIP 2223123. Well what’s been happening is this: these processes would check Calypso and determine it that Gosh-darn-it… we ain’t got that Bond definition. Let’s go fetch it!

5 processes on 5 different machines each generating a request file, cy_2223123.req. Some time passes and the Bloomberg Engine running on each box processes the file and uploads it to the remote (COMMON!) Bloomberg directory. Some time later, these 5 processes would each look for some file, cy_2223123.out, download it back locally and proceed to delete the remote file.

Hmmm…. can I get semaphores implemented on top of FTP? On second thought, don’t answer that…

I’m surprised that the process failed so infrequently, but occasionally, you’d have some poor process who’d never get its .out file. No wonder, some selfish sibling would have wiped it out before he got a chance to retrieve it. The funny thing is that, had I even pondered the question back at Calypso, it would have easily been apparent. I just never bothered to think: “Hey, wait… what if they run this process 5 times instead of just once, which seems logical.”

The quick fix? We’ve decided, until Calypso issues an official fix, to stagger the execution time for OTR Processing. Typically TIBCO will broadcast OTR information sometime between 3 and 4 AM and, of course, that information remains cached for new subscribers. So as long as we don’t connect before 4 AM and we have each box start processing OTR Bond messages on TIBCO at 10 minute interval, each instance of Bloomberg Engine should, hopefully, get to do its thing by itself. It’s not the most elegant solution, but hopefully it’ll do the trick!

Bloomberg, SWIFT, and other dinosaurs

Why are some of the most widely used protocols and data streams on Wall Street so antiquated? I’ve been working recently to interface my employer’s product with Bloomberg security data. FTP? Flat-file format? Errrr… I’m surprised they didn’t ask me to use punch cards to send my request via US Mail.

Now my first thought was that our client (on whose behalf I’m implementing this interface) is too cheap to get an upgrade on its Bloomberg License. Still, that doesn’t really make sense when upgrading to real-time quotes and reference data updates would have a significant upside for its traders.

No… I think this is all Bloomberg’s doing. They’ve got antiquated software that’s clearly (as far as I could discern from the API documentation) running batch jobs to query their data feeds and… well, you know… concurrency is, like, really hard. 😉 Then again, it could simply be that, like Microsoft and other pseudo-monopolies out there, Bloomberg knows he’s got the best data on the street and his clients will do just about anything to get to it. Clearly, if indices, quotes and credit events were drugs, Bloomberg would be the Pablo Escobar of the digital age.