Profile Picture

Building a Streaming Data Plugin for Edition 2

Posted By steveo311 8 Years Ago
Message
Posted Friday February 20 2009
I have spent the last several days implementing a plugin to stream data and thought I would share my initial experiences to see if anyone has any comments or suggestions. First and foremost the level of support I received during this task was amazing and has led me to select RE as the platform to move ahead with! There are about half a dozen ats platforms that support portfolio trading on the market and after six weeks of hard labor I realized that implementing my strategy on any one of them was going to take significant customization. Since writing a custom data feed plugin (needed by all the platforms I looked at) is usually not well documented or even possible, I consider support to be the key differentiator.

In order for me to implement my strategy in RE I needed to offload the tick -> bar processing by having a separate process do the conversion then stream the data through a plugin. Essentially this results in a reduction in the number of ticks RE needs to process to 4 / bar / symbol. I.e. each bar can be represented by 4 ticks, 1 each for open, high, low, and close. The overall proces uses QCollector to converts raw tick streams to bars then the plugin reads the bars and coverts each one to 4 ticks and feeds them to RE for conversion back to bars.

Fortunately QCollector does most of the heavy lifting out of the box and is compatible with both esignal or iqfeed. It writes all data to text files so debugging is simplified as well. I implemented a plugin that reads the QCollector files and creates the necessary ticks in about 2 days (granted they were rather long days). Although QCollector supports a windows messages interface for notification of "new data ready" my first implementation uses the last file write timestamp to keep track of when new bars. I did some research but could not see how to receive windows messages from C# in a DLL. If there are any .NET guru's out there, I would love to hear from you!!!

I am currently in early testing and the system seems to work well in live mode with 1 minute bars (i.e. just monitoring symbols). The data looks accurate and load on RE has been significantly reduced. I am however having a problem when I try to run a strategy (just using the sample cross over). I get the following error:

Not expecting a partial bar at this point.
   at RightEdge.Common.SymbolScriptCollection`1.NewBar()
   at MySystemBase.NewBar() in c:\Documents and Settings\steveo\Application Data\Yye Software\RightEdge\2008.2.0.0\Samples\Sample Systems\CrossOver\C#\BaseClasses.cs:line 31
   at RightEdge.Common.Internal.SystemRunner.x29c1ac0b845a9fb3(FrequencyNewBarEventArgs xce8d8c7e3c2c2426)
   at RightEdge.Common.Internal.SystemRunner.xb0b1947cd5228dff(IEnumerable`1 xcbd2c342d7cebcc1)
   at RightEdge.Common.FrequencyManager.xc27d8c8837babbfa()
   at RightEdge.Common.FrequencyManager.xfc18b789d02886f4(DateTime xb21f13a9707ad954)
   at RightEdge.Common.FrequencyManager.ProcessTick(Symbol symbol, TickData tick)
   at RightEdge.Common.Internal.SystemRunner._x4d2abf28cf2498fd(Object xe0292b9ed559da7d, NewTickEventArgs xfbf34718e704c6bc)
   at RightEdge.Common.TickGenerator.ProcessTick(NewTickEventArgs args)
   at RightEdge.Common.Internal.SystemRunner.ProcessTick(Symbol symbol, TickData tick)
   at RightEdge.Shared.SystemWrapper.ProcessTicks(List`1 ticks, Boolean moreTicksPending)
   at RightEdge.Shared.SystemWrapper.ProcessTicks(List`1 ticks, Boolean moreTicksPending)
   at RightEdge.Shared.TradingModuleWrapper.ProcessTicks(List`1 ticks, Boolean moreTicksPending)
   at RightEdge.LiveSystemwindow.xf8e43e374a2dae4c()

If anyone has any thoughts on why this might be, I would love to hear that too. Also, if you have any thoughts on the overall scheme I would love to hear those as well.

Thanks in advance

Steve

Posted Friday February 20 2009
I did some research but could not see how to receive windows messages from C# in a DLL

Hi Steve,
These may help with your windows messaging issue.
http://www.codeproject.com/KB/dotnet/devicevolumemonitor.aspx
http://www.codeproject.com/KB/system/WilsonSystemGlobalHooks.aspx

Cheers!
Mark
Posted Saturday February 21 2009
Thanks a lot for the reply Mark. My problem was that I could not see a way to overload winproc and catch the events? I.e. I dont think I have the necessary access since its a plugin? Am I missing something or would you know if there is a work around?

My current solution was to write a separate forms based app that translates from windows messages to sockets.

Posted Saturday February 21 2009
This is the hack method from the "global hooks" link.
GetProp and SetProp are API functions that allow you to associate any type of handle with any window. That is, if you have a window handle (hWnd), you can associate any other handle with it, and in such a way that any application can access that data. Luckily, there's one window handle that's essentially constant in any Windows session, the desktop handle.

It does seem to only work for 1 trapping application at a time though. If you need more than one instance of RightEdge to receive your data then I would probably just stick with your socket app.

Cheers!
Mark
phg
Supreme Being (44,911 reputation)Supreme Being (44,911 reputation)Supreme Being (44,911 reputation)Supreme Being (44,911 reputation)Supreme Being (44,911 reputation)Supreme Being (44,911 reputation)Supreme Being (44,911 reputation)Supreme Being (44,911 reputation)Supreme Being (44,911 reputation)
Posted Saturday February 21 2009
I'm not sure, but is this reference to baretail relevant? http://baremetalsoft.com/baretail/

-Pete

See also Yahoo group about applying RE.

Posted Saturday February 21 2009
Thanks for sharing this, Steve.  I think a lot of folks customize and the more information out there, the better.  As far as your exception, can you dump out the data to a text file before it is passed to RE so that I could have a look?  Is this happening if you watch a single symbol only?

steveo311 (2/20/2009)
I have spent the last several days implementing a plugin to stream data and thought I would share my initial experiences to see if anyone has any comments or suggestions. First and foremost the level of support I received during this task was amazing and has led me to select RE as the platform to move ahead with! There are about half a dozen ats platforms that support portfolio trading on the market and after six weeks of hard labor I realized that implementing my strategy on any one of them was going to take significant customization. Since writing a custom data feed plugin (needed by all the platforms I looked at) is usually not well documented or even possible, I consider support to be the key differentiator.

In order for me to implement my strategy in RE I needed to offload the tick -> bar processing by having a separate process do the conversion then stream the data through a plugin. Essentially this results in a reduction in the number of ticks RE needs to process to 4 / bar / symbol. I.e. each bar can be represented by 4 ticks, 1 each for open, high, low, and close. The overall proces uses QCollector to converts raw tick streams to bars then the plugin reads the bars and coverts each one to 4 ticks and feeds them to RE for conversion back to bars.

Fortunately QCollector does most of the heavy lifting out of the box and is compatible with both esignal or iqfeed. It writes all data to text files so debugging is simplified as well. I implemented a plugin that reads the QCollector files and creates the necessary ticks in about 2 days (granted they were rather long days). Although QCollector supports a windows messages interface for notification of "new data ready" my first implementation uses the last file write timestamp to keep track of when new bars. I did some research but could not see how to receive windows messages from C# in a DLL. If there are any .NET guru's out there, I would love to hear from you!!!

I am currently in early testing and the system seems to work well in live mode with 1 minute bars (i.e. just monitoring symbols). The data looks accurate and load on RE has been significantly reduced. I am however having a problem when I try to run a strategy (just using the sample cross over). I get the following error:

Not expecting a partial bar at this point.
   at RightEdge.Common.SymbolScriptCollection`1.NewBar()
   at MySystemBase.NewBar() in c:\Documents and Settings\steveo\Application Data\Yye Software\RightEdge\2008.2.0.0\Samples\Sample Systems\CrossOver\C#\BaseClasses.cs:line 31
   at RightEdge.Common.Internal.SystemRunner.x29c1ac0b845a9fb3(FrequencyNewBarEventArgs xce8d8c7e3c2c2426)
   at RightEdge.Common.Internal.SystemRunner.xb0b1947cd5228dff(IEnumerable`1 xcbd2c342d7cebcc1)
   at RightEdge.Common.FrequencyManager.xc27d8c8837babbfa()
   at RightEdge.Common.FrequencyManager.xfc18b789d02886f4(DateTime xb21f13a9707ad954)
   at RightEdge.Common.FrequencyManager.ProcessTick(Symbol symbol, TickData tick)
   at RightEdge.Common.Internal.SystemRunner._x4d2abf28cf2498fd(Object xe0292b9ed559da7d, NewTickEventArgs xfbf34718e704c6bc)
   at RightEdge.Common.TickGenerator.ProcessTick(NewTickEventArgs args)
   at RightEdge.Common.Internal.SystemRunner.ProcessTick(Symbol symbol, TickData tick)
   at RightEdge.Shared.SystemWrapper.ProcessTicks(List`1 ticks, Boolean moreTicksPending)
   at RightEdge.Shared.SystemWrapper.ProcessTicks(List`1 ticks, Boolean moreTicksPending)
   at RightEdge.Shared.TradingModuleWrapper.ProcessTicks(List`1 ticks, Boolean moreTicksPending)
   at RightEdge.LiveSystemwindow.xf8e43e374a2dae4c()

If anyone has any thoughts on why this might be, I would love to hear that too. Also, if you have any thoughts on the overall scheme I would love to hear those as well.

Thanks in advance

Steve

Posted Saturday February 21 2009
Thanks for the reply Bill and for the offer to take a look at the data! I was working on another task and needed to clean the database yesterday so I don't have much right now. I'll get some more live data Monday morning and send it over.
Posted Saturday February 21 2009
The "not expecting partial bar" exception is a known bug in the RE2008E2 beta.  I hope to release a fix for it soon.

Thanks,
Daniel



Similar Topics


Reading This Topic


2005-2017 © RightEdge Systems