RightEdge Forums / Programming and Software Development / Programming Questions / Memory issues - another thread / Latest Posts

RE: Memory issues - another thread

dplaisted — Mon, 29 Mar 2010 01:37:33 GMT

I've improved the equity curve's performance. The chart control was apparently getting bogged down when trying to draw the equity curve with too many data points. I've limited the number of data points to 1000. It still takes up a lot of CPU time when you move your mouse over the chart (because it keeps redrawing it), but I think it will be much improved. This will be in the next build, so once we release that, try it out and let us know.

Thanks,
Daniel

RE: Memory issues - another thread

kaizen — Tue, 16 Mar 2010 18:53:09 GMT

One other thing to add to the list to check is when you start a sim and the data starts loading then you cancel the simulation before it finishes.

When doing this have seen it hold the memory that it had been using.

RE: Memory issues - another thread

billb — Tue, 16 Mar 2010 16:03:05 GMT

Too many data points and zooming are two possible causes, but we are also looking into ways to make it faster on our end. Approximately how many bars is your simulation?

RE: Memory issues - another thread

PBJ — Tue, 16 Mar 2010 09:38:45 GMT

Ok Daniel thanks.
I must bring to your attention that it's hard for a new comer like me to test the software fully before purchasing if I cannot display an equity curve.

Do you have an idea what triggers the freeze (and 100% CPU usage) when the Equity Curve tab is clicked? The graphic component maybe? An infinite loop?

A little more information on my configuration:
OS : XP Home Edition SP3 32 bits FR
RAM : 2.5 GB
CPU : Pentium 4 @3.00 GHz
.Net Framework: [i]currently "repairing" 3.5 SP1[/i] (just to see if the problem comes from it)
RightEdge: Edition 2010 Build 17 - free 45 day trial
System: my first trading system tentative, running simulation on C (Citigroup stock), daily bars from Yahoo! historical data services (included in RE), ~8377 daily bars

Let me know if I can be of further assistance. I'd be pleased to help you help me.

Best,
PBJ

RE: Memory issues - another thread

dplaisted — Mon, 15 Mar 2010 23:25:16 GMT

The system results memory leak should be fixed in the next build.

We still need to fix the equity curve issue, that may or may not make it into the next build.

Thanks,
Daniel

RE: Memory issues - another thread

PBJ — Mon, 15 Mar 2010 12:58:52 GMT

Hi,

I'm currently reviewing RE 2010 using the free trial option of 45 days (great stuff!).
I have coded and tested my first trading system and have the exact same problem: when the simulation is complete and I click the Equity Curve tab, the app freezes and uses 100% of 1 core on my Dual Core CPU.

Can I help you identify the problem? The only thing I did was to click Equity Curve.
Running latest version of RE 2010 on XP Home.
Please let me know if I can be of further assistance.

Best,
PBJ

RE: Memory issues - another thread

dplaisted — Wed, 10 Mar 2010 13:59:53 GMT

I have made some progress on identifying the memory leak and expect to have it fixed in the next build.

Thanks,
Daniel

RE: Memory issues - another thread

bbmat173 — Tue, 09 Feb 2010 00:19:16 GMT

Bill,

the big issue remains the memory that is not released after each backtest. Can you please try to give this priority. Seriously, back tests cannot be run efficiently if memory is not released after each run. I dont try to bitch about it (though I also dont wanna give up on this issue) but ultimately you make it easy for users to move onto their own VS platform, hook up to a HDF database or use the open source Tree Data Server and vola, you have a back testing platform that may be crude but at least do the things you want. Issue is, it take some time to develop which is why I am still with RE. But I got to the point where I do all my optimizations in matlab and currently work to code a real-time live strategy platform in VS for live trading because of limitations in RE. You guys came up with a great product, and I think its still the best ouf of the pack, and I truly want this product to succeed. Memory issues are for me the number one reason right now to look at alternatives...I dont think it has to be that way...which would be a win...win for all of us.

[quote][b]billb (2/8/2010)[/b][hr]I've reproduced the equity curve problem. It's simply getting bogged down by the data points and the zoom feature is exacerbating this. I'm investigating ways to bind more efficiently, but may have to turn off zooming (or provide the option) to get this thing acting right.[/quote]

RE: Memory issues - another thread

billb — Mon, 08 Feb 2010 12:24:45 GMT

I've reproduced the equity curve problem. It's simply getting bogged down by the data points and the zoom feature is exacerbating this. I'm investigating ways to bind more efficiently, but may have to turn off zooming (or provide the option) to get this thing acting right.

RE: Memory issues - another thread

gtomez — Sun, 07 Feb 2010 11:40:07 GMT

setting SystemStatistics.Enabled = false each simulation run adds 20 MB of memory usage, set to false ech run adds 300-400 MB.

My System generates about 400 Trades. I could do all statistics in Excel using 400KB ...

Could anybody give me an example code how to export position/trade list to a CSV plus adding properties (data start date, allocation, Max open etc.) to this output?

RE: Memory issues - another thread

gtomez — Sun, 07 Feb 2010 09:24:07 GMT

[quote][b]bbmat173 (2/6/2010)[/b][hr]I use NLOG to output a lot of stats[hr][/quote]

Hi bbmat!

Could you please post some examples of how to use NLOG here http://www.rightedgesystems.com/forums/Topic9946-18-1.aspx?Highlight=NLOG

Hopefully we will get soon some statements from RE to these memory issues...

Regards,

RE: Memory issues - another thread

bbmat173 — Sat, 06 Feb 2010 03:34:49 GMT

Having the exact same problems, so far nothing has been done to solve those bugs. Generating the equity curve takes very long time so I kinda gave up on it and do it myself (I use NLOG to output a lot of stats). However, RE is kind of getting useless if I cannot run a simple optimization or more than 1-2 backtests without having to close down the whole application, reload it, reload my strategy, re-selecting all symbols for the backtest, and run the next test....seriously, we started coding our own stuff in VS C#....if those issues cannot be solved soon then running backtests in RE is more a hassle than anything else. Quite disappointed about this because sometimes menial tasks are given priority while memory issues are piling up but nothing is done about it. I pointed it out several times among other users...

I mean, isnt that a CLEAR BUG that memory is not released even after closing down the results tab, clearing output (I dont generate output in the first place). It should not matter how many symbols I use, how many userseries and so forth, point is, when I want to run another backtest I should be able to close whatever needs to be closed in order to re-capture memory...

[quote][b]gtomez (2/4/2010)[/b][hr]Hi!

I have some problems using RE as I cannot run simulations only once ore twice until I get an OutOfMemory Exception.

At startup of RE it uses app. 150 MB of memory

After the first simulation it uses 800 MB and wont free up the memory afterwards. Second simulation pushes up memory usage to 1.4 GB.

I'm generating UserSeries for about 400K Bars.

Is there a way to set free the used memory after simulation?

If I'm not interested in displaying UserSeries after the simulation, is it possible to use MaxLookBack for UserSeries as well?

Increasing RAM from 4GB to 8 GB would allow 4 simulation runs but wont solve my problem ;)

Kind Regards,

GT[/quote]

RE: Memory issues - another thread

gtomez — Fri, 05 Feb 2010 09:18:57 GMT

[quote][b]billb (2/5/2010)[/b][hr]You're running the charts on 400,000 data points?
[/quote]

Yes, after consultation of my pocket calc it could be even more - up to 600-800k I guess
Running 2 Simulations is possible, but not another or displaying equity curve.

RE: Memory issues - another thread

billb — Fri, 05 Feb 2010 08:55:14 GMT

You're running the charts on 400,000 data points?

[quote][b]gtomez (2/5/2010)[/b][hr]Sometimes I close it sometimes not - seems to make no difference (long, short, b&h - reports are disabled)

In the output window are about 30 entries concerning MaxOpenPositions, Used Memory and Counter (the latter two have one line each).

Another problem showed up yesterday night:
When I chose "Equity curve" to be evaluated in the reports, the application freezes still using one CPU 100%. Memory usage does not change. Even after 15 min the application is still "frozen".

RE Ed2 build 14
Win7 64 bit
AMD Phenom Quad
4 GB memory[/quote]

RE: Memory issues - another thread

gtomez — Fri, 05 Feb 2010 05:51:06 GMT

Sometimes I close it sometimes not - seems to make no difference (long, short, b&h - reports are disabled)

In the output window are about 30 entries concerning MaxOpenPositions, Used Memory and Counter (the latter two have one line each).

Another problem showed up yesterday night:
When I chose "Equity curve" to be evaluated in the reports, the application freezes still using one CPU 100%. Memory usage does not change. Even after 15 min the application is still "frozen".

RE Ed2 build 14
Win7 64 bit
AMD Phenom Quad
4 GB memory

RE: Memory issues - another thread

dplaisted — Fri, 05 Feb 2010 01:12:28 GMT

Are you closing the results window after each run or leaving it open?

Is your system generating a lot of output (to the output window)?

Memory issues - another thread

gtomez — Thu, 04 Feb 2010 16:13:33 GMT

Hi!

I have some problems using RE as I cannot run simulations only once ore twice until I get an OutOfMemory Exception.

At startup of RE it uses app. 150 MB of memory

After the first simulation it uses 800 MB and wont free up the memory afterwards. Second simulation pushes up memory usage to 1.4 GB.

I'm generating UserSeries for about 400K Bars.

Is there a way to set free the used memory after simulation?

If I'm not interested in displaying UserSeries after the simulation, is it possible to use MaxLookBack for UserSeries as well?

Increasing RAM from 4GB to 8 GB would allow 4 simulation runs but wont solve my problem ;)

Kind Regards,