I came upon an interesting problem late last week.
I’ve recently implemented an OTR Bond interface to Calypso so that, when new TIBCO messages alerts us of a new “On-the-run” Bond issue, we can insert the appropriate data into Calypso. Of course, if it’s a new Bond, there’s also a good chance that we don’t yet have the Bond definition in Calypso and, for that, we use the Bloomberg Connect API to retrieve the Bond and link it to the Calypso BondBenchmark… Sorry. That was just an overview but I can already see that most of y’all have your eyes all glazed over. Didn’t mean to bore you to death.
Anyway, I’m rather familiar with Calypso’s Bloomberg Connect API and how it works. Puh-lease… give me a hard one!!! After all, I pretty much wrote the whole thing single-handedly a couple of years ago. Let’s just say that a request file is generated, then FTP’ed over to Bloomberg. Then, it’s your responsibility to poll the remote directory for the equivalent response file. Once it is available, you download it and parse it locally with the data you requested. Pretty straightforward, right?
Well, when I implemented the Bloomberg interface, I thought of the “requested” scenario and never bothered (foolish me!) to think outside the box. Let’s say you request a Bond with CUSIP 123456, then I’d simply generate a request file called cy_123456.req (cy being short for Calypso. Y’see? To top it all off Bloomberg states that there’s a maximum number of characters in the file name. Some puny amount like 12 or 15, I think.) So anyway, cy_123456.req would get processed and FTP’ed to the remote directory. Then, I’d poll every once in a while for a file called cy_123456.out. When the file would be present, I’d FTP it back to my local directory and remove the file remotely, thus cleaning up after myself.
This ain’t rocket science, right? That should work just fine, no?
If you’ve got one instance of Bloomberg Connect running, it runs fine and dandy, yes… but here at Countrywide, we got a bunch of environments all running in parallel. We’ve got a couple of development boxes, UAT, and of course, the production environment. The configuration for these boxes are identical and the OTR Bond adapter task kicks off from 4 AM until 5:30 AM. So now you have these 5 or 6 processes all running on different boxes each listening to TIBCO. Let’s say at 4:05 TIBCO broadcasts: The 3 month OTR Bond has CUSIP 2223123. Well what’s been happening is this: these processes would check Calypso and determine it that Gosh-darn-it… we ain’t got that Bond definition. Let’s go fetch it!
5 processes on 5 different machines each generating a request file, cy_2223123.req. Some time passes and the Bloomberg Engine running on each box processes the file and uploads it to the remote (COMMON!) Bloomberg directory. Some time later, these 5 processes would each look for some file, cy_2223123.out, download it back locally and proceed to delete the remote file.
Hmmm…. can I get semaphores implemented on top of FTP? On second thought, don’t answer that…
I’m surprised that the process failed so infrequently, but occasionally, you’d have some poor process who’d never get its .out file. No wonder, some selfish sibling would have wiped it out before he got a chance to retrieve it. The funny thing is that, had I even pondered the question back at Calypso, it would have easily been apparent. I just never bothered to think: “Hey, wait… what if they run this process 5 times instead of just once, which seems logical.”
The quick fix? We’ve decided, until Calypso issues an official fix, to stagger the execution time for OTR Processing. Typically TIBCO will broadcast OTR information sometime between 3 and 4 AM and, of course, that information remains cached for new subscribers. So as long as we don’t connect before 4 AM and we have each box start processing OTR Bond messages on TIBCO at 10 minute interval, each instance of Bloomberg Engine should, hopefully, get to do its thing by itself. It’s not the most elegant solution, but hopefully it’ll do the trick!