 Posted 4/1/2011 01:38:34
|
|
|
|
Is there a way to aggregate bars during simulation to reduce the memory required? For example, I have 1 min data which I run a daily system against; however, RE appears to load all minute data even when the Frequency (under General properties) is set to Daily. This obviously results in a larger memory footprint, which additionally reduces the number of symbols I can execute against, and a much slower simulation.
Thanks, Duane
|
|
|
Posted 4/1/2011 16:42:16
|
|
|
|
If the above is not possible by simple means, I can always develop a DataStorage plugin to aggregate. I should note that I actually need aggregation from minute to hourly data even though the system is run on a daily basis.
For the DataStorage plugin I noticed a BarAggregator.AggregateBars method which works nicely; however, the method is obsolete. Is there a different way that I should accomplish aggregation?
Note, the preferred method to aggregate data would be in the symbol script to avoid duplication of data in the db.
Thanks, Duane
|
|
|
Posted 4/4/2011 14:01:56
|
|
|
|
RE appears to load all minute data even when the Frequency (under General properties) is set to Daily.
Well, to convert the minute data to daily data, it has to read all of the minute data, which means it all gets loaded into memory at one point or the other. However, it doesn't need to load it all at once. RightEdge does it in batches of 5000 bars per symbol. In .NET, memory that isn't needed anymore isn't necessarily garbage collected immediately. So the memory usage of a process isn't always a good indication of how the program is actually using memory-- if it's not actually running out of memory.
RightEdge breaks each of the bars from the data source into ticks. The ticks are then re-aggregated to the frequencies requested by the system. The ticks are also sent to the system as events. So using higher frequency data will be slower because there are more ticks to process. If you aggregate it into hourly data, the simulation will be faster, but the accuracy of the simulation will be less because there are only 1/60th as many price data points.
I hope this helps.
Thanks,
Daniel
|
|
|
Posted 4/4/2011 16:09:35
|
|
|
|
The memory usage is too much on a minute basis so I converted to hourly data which will work. In order to do so, I’m using BarAggregator.AggregateBars but noticed that this is method is obsolete. Can you elaborate on why this is obsolete? Also, is there a different method to accomplish the same?
Thanks, Duane
|
|
|
Posted 4/6/2011 01:04:35
|
|
|
|
The old AggregateBars method was from before we added the multiple frequency support. The new way to do it would be to create a TimeFrequency and feed it the bar data. There's a BarUtils.ProcessRListInBarGenerator method which makes this a bit easier. Here's how you can use it:RList<BarData> AggregateBars(Symbol symbol, RList<BarData> sourceData, DateTime lastBarEnd, FrequencyPlugin targetFrequency)
{
RList<BarData> ret = new RList<BarData>();
IFrequencyGenerator generator = targetFrequency.CreateFrequencyGenerator();
// Bar construction type isn't important because we are aggregating bars, not ticks
generator.Initialize(symbol, BarConstructionType.Default);
generator.NewBar += delegate(object sender, SingleBarEventArgs e)
{
ret.Add(e.Bar);
};
generator.NewTick += delegate(object sender, NewTickEventArgs e)
{
ret.PartialItem = e.PartialBar;
};
BarUtils.ProcessRListInBarGenerator(generator, symbol, sourceData, lastBarEnd);
return ret;
} You say the memory usage is too much. What exactly is happening? Is it throwing out of memory exceptions, running to slow, or what? Thanks,
Daniel
|
|
|
Posted 4/6/2011 01:44:50
|
|
|
|
Thank you for the code. Regarding the memory consumption, 2 things are occurring: the combination of RE and SQL server memory consumes a large amount of memory and the simulation runs slow due to the amount of minute bar data that needs to be processed (> 30 minutes for a 4 year sim on 1000 symbols).
I appreciate your support.
Duane
|
|
|
|