Profile Picture

Memory, optimization, and other suggestions

Posted By jkhax0r 8 Years Ago
Message
Posted Sunday July 12 2009
I am evaluating the new beta release 12.

A few suggestions I have found...

===1===

I am running an optimization on a relatively simple strategy over 1 month of test data (1-minute bars, only EUR/USD), and I am getting out of memory exceptions after only about 30 runs.

Looking at a plot of the memory used by RE during optimization, it just keeps going up and up with each optimization run.

Is this how its supposed to behave? My system has 1.5GB of memory, but even with 4GB I could barely even run 100 optimization runs. It seems you could save the results of every pass to file and use much less memory.

===2===

It is somewhat irritating that you can't see the results of the optimization in real-time. At least showing a non-interactive chart of runs vs profit or a list of each run and some basic stats would be worthwhile. Take a look at how MetaTrader does it. I usually run optimizations that take 2+ days over 1000+ passes, and then frequently will just stop the run and evaluate the results under MT. Rather than just pressing Cancel, you should just have a "Stop" that then show the results of the passes that did complete.

===3===

Similar to above... its kind of annoying that both the optimization and simulation progress indicators are modular and don't let you manipulate the program while running. Also, instead of canceling, if you could just abort but still see the results of what did finish, it would make the backtesting/optimization a lot more useful.

Overall RE impressed me a lot, but when I finally started running back-tests and optimizations, the slugishness and memory consumption has made it almost unusable. I really like the back-test results, but its just a pain that I have to reduce my symbols and time-frames just to have enough memory. BTW, I have MaxLookback set to 50.

Also, I will 2nd another users post - optimization parameters REALLY need to be saved from run-to-run, at a bare-minimum. Especially because I must resort to only running 20 optimization passes at a time.


Posted Monday July 13 2009
Thank you for the suggestions. The memory issue has been quite elusive, but we've spent and continue to spend lots of resources tracking it down. We will definitely report any forward progress on that issue. The other suggestions have been logged.
Posted Monday July 13 2009
Something that seems odd to me is that when I close back-test results and the respective graphs, the memory that was loaded and allocated from that back test never frees. I periodically just close RE and reload to kind of reset that b/c after 3 or 4 back-tests I am running out of memory again.

I have learned to deal with this, and plan on just getting more memory (and a 64-bit OS) if I end up going farther with your software. The other 2 issues I think are just as important though (Abort optos and back-tests and still see progress that was made, as well as monitor progress during opto/back-test). Just a simple enhancement to the modular progress dialogs might be enough if its too much effort to make then non-modular.

The reason for these requests is because often during development I will try something out, and if it is failing miserably, theres no reason to continue the opto or back-test.

Great work overall. I think your software has much more potential than others, its just a few major issues that I'm sure you guys will have straightened out in the next few months.
Posted Sunday July 26 2009
The next release will save you optimization parameter settings, so you won't have to reenter the high/low/number of steps each time if they don't change.

I've also fixed the memory leak that was occurring when you closed the system results and the memory wasn't getting freed.

The memory leak that occurs while running the actual optimization is more elusive. RightEdge does save the results of each parameter combination to disk after it finishes running it, and the memory should be free to be collected. And that is what happens when I run optimizations. I was able to reproduce the problem once, but that was a while ago and I haven't had any luck since then. It may depend even on the OS you are running. I am using the Windows 7 RC right now, what OS are you running?

Thanks,
Daniel
Posted Sunday July 26 2009
Great news on these memory related fixes.

I am running Windows XP 32-bit w/ all updates and Ed1 384 of RE. I am using one minute bar data on 6 Forex currency pairs and roughly 3-6 months of data during optimization will cause the memory exception. It seems like it just continues to grow until the exception - I usually only get about 10 runs done. I have 2.5GB of Memory but the memory exceptions always happens at about 1.5GB-2GB in use by RE, which probably makes sense considering that DLLs and code probably fill about 2GB of address space as well. I may upgrade to 64-bit OS soon so I can get over this, but that would be unnecessary if this leak gets resolved.

BTW, I have noticed that as soon as the optimization finishes, the memory drops back down and then I can double click any run and it seems to do the right thing. I just have to get to the final run w/o running out of memory, which sucks becuase often I will see that its not going to make it and have to cancel (which is why being able to cancel and still see the finished run results would be nice). I will look into the issue and see if I can get you any info to help reproduce it.

Do you guys perhaps have an FTP site where I could drop my data sets, configuration, strategy? Would this be helpful to you to maybe reproduce the issue - I know reproduction is usually 90% of the battle?
Posted Monday July 27 2009
jkhax0r (7/26/2009)


Do you guys perhaps have an FTP site where I could drop my data sets, configuration, strategy? Would this be helpful to you to maybe reproduce the issue - I know reproduction is usually 90% of the battle?


You are absolutely right. Reproducing the problem is indeed 90% of the battle most of the time. And yes, we can definitely get you some ftp access to send across some big files. Please send an email to resupport at rightedgesystems referencing this post and someone will get an account setup for you.
Posted Friday August 07 2009
I was able to reproduce the issue with the system and data you uploaded. Thanks for doing so!

In .NET in general, when memory isn't getting released it's usually because there is still a reference to it from some active object. However, it can be hard to figure out where that is happening. There are two cases in your system where it leaves a reference to your system active. I am modifying RightEdge so that each run within an optimization will take place in a separate AppDomain, which should mean you won't have to worry about this. Until we release a new build though, here is what is happening and how you can fix it:

In your helper class you are creating a timer, which calls an event in your helper class after 1 hour. Through the event, the timer has a reference to your helper class, which has a reference to your system class. It looks like this is only meant for live mode, so I would recommend not creating the timer if not in live mode.

You are overriding the Shutdown method in your MySystem class. It turns out that the base class startup modifies the default console output stream to send to your system, and in the base Shutdown method this is unhooked. So you need to call the base version of the Shutdown method from within your overridden version.

Thanks,
Daniel


Similar Topics


Reading This Topic


2005-2017 © RightEdge Systems