Profile Picture

Advanced Topics

Posted By erockenbach 8 Years Ago
Posted Sunday October 10 2010
Dear RE staff,

I was about to code a few changes to get the most out of RE and I had a few questions before I began. 

Number one I was going to modify the optimization plugin of RE to support multithreading so in other words a thread pool would be used to run through the configuration scenarios through the optimization plugin.  I wanted to verify that nothing is setup as a singleton in the core or paperbroker code that would prevent this from working properly.

Number two is I notice RE can consume a lot of ram when running longer scenarios my guess is that the bar data from whatever frequency you subscribe to is being stored in memory.  I was planning on implementing a flushing mechanism by eleminating bar data no longer needed.  Is this a correct statement, is this what is consuming the memory?  Also do I need to subsribe to all frequencies and flush them all or only the frequencies that my simulation is set to run on?

Number three is I plan to implement a large SQL 08 R2 cluster and have a considerable amount of RE instances on various machines connecting to RE to pull in the SAME data across different servers.  This should be fine I suspect, but what about the fact that when hooked up to a live account I might have 5 instances of RE running on 5 different machines connecting to one cluster for example all trading the same symbol.  They will all be trying to write the data they are recieving from the live brokerage account to SQL correct?  Will this work?  Wouldn't they be trying to right the exact same data five times if there were 5 instances of RE running the same symbol?

Number four..  I haven't really researched this, but is it possible to write plugins for the Win32 RE interface so for example additional menu items, etc where I could provide custom information?

Thanks in advance for the advice!

Edited: Sunday October 10 2010 by erockenbach
Posted Sunday October 10 2010
There shouldn't be any singletons that would prevent you from multithreading, and on top of that each time your optimization plugin calls RunSystem a separate AppDomain is created for that run, so even if your trading system or one of your plugins used singletons, they would still be isolated from other copies of your system running on other threads.

For controlling RightEdge's memory usage, see the "Large Data Set Support" section under the "What's New In This Edition?" help topic in the RightEdge documentation. I don't think there's any way you could implement a flushing mechanism yourself, but you can let RightEdge do it by setting the MaxLookBack property of the bar lists. System statistics and indicators don't have a way of flushing old data. See the help topic for some tips on how to reduce their memory usage if you need to.

I would recommend only saving the data from one of the computers. In the watchlist folder settings, you can specify whether realtime tick and bar data should be saved to the data store. Turn it off for all of the computers except one. Note that the SQL Server data storage plugin code is available, so if you do need to do something fancy you should be able to modify it to your needs.

We don't have a way of writing plugins that add commands to the RightEdge UI itself. If what you want is to have some manual control while running a live system, the system can display its own window for that (which won't be integrated with the RightEdge UI). You will need to create a separate UI thread for the system's window, but you should only access RightEdge APIs from the system thread, so it is a bit tricky. There is a sample system which shows the basics of how to do it though.

Posted Monday October 11 2010
Excellent thanks for the quick reply.  Great product btw and I appreciate the continued enhancements. 
Posted Tuesday November 16 2010
Daniel - What if one has multiple instances on one PC. It doesn't seem possible to specify that only one to write data since they both share a common SymbolConfig.xml

Is that correct?

Using the local datastore, I find that there are times (once a day) when I get duplicate bars if I have two instances running on the same PC. I have to use the data editor to fix the data for most of the symbols every morning before a restart. I ended up only running one instance per machine but that obviously isn't ideal....

If there is some way to get only one instance on the same PC to write data would be very beneficial (in terms of extra PCs /screens and RE licenses).


Edited: Tuesday November 16 2010 by smersh
Posted Saturday November 20 2010
You could set up two watchlist folders, identical except for the fact that one is set to save live data, and one isn't. Then you'd need to make sure to only use the folder that saves the data with one RightEdge instance.

Another option, which would be more effort initially but easier to use on a day-to-day basis, would be to disable saving live data within RightEdge, and have an external program which subscribes to the live data and saves it to the data store. The source code to the data store and most of the data plugins is available, so this shouldn't be too hard.

Posted Saturday March 26 2011
dplaisted (10/10/2010)
There shouldn't be any singletons that would prevent you from multithreading, and on top of that each time your optimization plugin calls RunSystem a separate AppDomain is created for that run, so even if your trading system or one of your plugins used singletons, they would still be isolated from other copies of your system running on other threads.


Hi Daniel, it's a good insight that the calls to RunSystem are executed on different AppDomains, but the problem I have with trying to thread the default Optimization plug-in as an exercise is that I depend on the instance of SystemRunSettings that are passed to the plug-in by RE. SystemRunSettings has no copy constructor, it has no Clone or MemberwiseClone methods. This means that the same instance of SystemRunSettings is passed to every call to RunSystem. Needless to say they all collide when executed simultaneously, so only the last parameters set by runSettings.SystemParameters actually get used for a full optimization run. I see no way of decoupling from this (due to aforementioned lack of methods to create a new instance with all the same properties).

Am I missing some solution, or is it actually impossible to run multiple calls to RunSystem at the same time? The SystemRunSettings type is serializable, so I suppose it is possible to serialize the given instance to a MemoryStream then deserialize into a new instance... but that seem like way to much work just to replicate some properties. Also, I'm not even sure the new instance would be valid, I mean the original SystemRunSettings instance comes from somewhere else in RE, so RE could have other references to it somewhere (or even some internal properties, which I wouldn't see and wouldn't be able to replicate with this method), will my own hacked up copy even work?

Thank you for advice.

Edited: Saturday March 26 2011 by alexkopy
Posted Saturday March 26 2011
Good catch on the lack of a Clone method for SystemRunSettings. Cloning it via serialization is probably the easiest way to do it for you. It doesn't have any internal data and you should be able to call RunSystem with a new instance of the class that you've created, either by cloning via serialization or just newing up a copy and populating the properties yourself.


Similar Topics

Reading This Topic

2005-2018 © RightEdge Systems