Profile Picture

Combine Aggregated Data and imported history data?

Posted By raidsan 8 Years Ago
Message
Posted Sunday July 12 2009
I usually use M1 period to receive forex bars data from IB, and run multiple Strategy on D1, H4, H1, M15, M5 period,
There is problem that the M1 bar data count is limit, and aggregated D1, H4, ... bars data is too less for analysis,

Is it possible to import D1, H4,... history bar data when set the bar frequency to D1, H4, ... , then back to M1 frequency ,
and after that , when client call history bar data for high-level above M1 period , RE will use dynamic aggregated bars first,
if not exist, then it will instead to use exists imported data ?
phg
Supreme Being (46,873 reputation)Supreme Being (46,873 reputation)Supreme Being (46,873 reputation)Supreme Being (46,873 reputation)Supreme Being (46,873 reputation)Supreme Being (46,873 reputation)Supreme Being (46,873 reputation)Supreme Being (46,873 reputation)Supreme Being (46,873 reputation)
Posted Sunday July 12 2009
You may want to accumulate the amount of data you want offline, and then import it. One utility that will do that is http://www.tradingsoftwarelab.com/jtwsdump/. (I have not used it so cannot speak from experience. See the Freelance forum on the IB site.)

-Pete

See also Yahoo group about applying RE.


Edited: Tuesday March 21 2017 by phg
Posted Monday July 13 2009
Thank for the suggestion, but I don't need the IB data download tool.

What I not expected, if I set a symbol base frequency to min1(config the folder's bar frequency), although I have import H1 frequecy data before, when I change the chart's frequecy to H1, it can only get aggregated H1 bars time range as same as the M1 data's time range, and cannot use the imported H1 bars data anymore.

now I understand, the limit of the option ""Save Aggregated Bars" in symbol folder settings.
I think RE should support to save multiple aggregated frequency bars data.
Suppose I select the bar frequency to "1 Minute", I hope RE will provide "MN, WEEK, DAY, HOUR, 5Min" checkbox to support save the corresponding data bars in datastore.
when use chart to select frequency or run trading system on frequecy to process history bars, if it request 60min frequency bars data, RE should retrieve hour bars from datastore , instead of using 1min bars to generate 1 hour bars.

I think the solution above is more appropriate for the actual situation.
the data is more far from now, the more high level is required.
It is not necessary and wasted to use detail bar data to generate past high level bars.
I think this solution will improve the whole system's availability in data layer.
Although one trading-system will run on a specify data frequency, but the designer's ideal will often changed in research. If the datastore only save one frequecy data, it will not suit the changed data requirement. The disk store cose is cheap, isn't it? so, store more aggegate data will satisfy the requirement in the future, just like the data warehouse.
The work of import history data should only do the first time, from then on, the system will auto update real data and aggregate data, it will make things easier, as a basic layer, a comprehensive datastore is very important.

Edited: Monday July 13 2009 by raidsan
Posted Tuesday July 14 2009
We made a conscience decision not to do it that way. And it's not the disk space that's the issue, it's flexibility. You may want frequencies x,y,z today, but frequencies d,e,f later. We store in the lowest frequencies and leave it up to the frequency plugin to aggregate accordingly. This provides ultimate flexibility and also allows for non-time based frequencies to be created from the same data as well.

You're basically asking to 'cache' a frequency instead of having it computed on the fly. If this is a performance problem, it's probably quite possible for you to create a data store that would save to all of your desired frequencies each time a tick or bar is committed. Then when frequency x is asked of the data store, you could load it directly from your custom data store.
Posted Tuesday July 14 2009
It is easy to save aggregate data in datastore plugin, but the IDataStore interface could spport to work in this way?
I have use mysql plugin to run a tradingsystem under sixtyminutes frequency on symbols which use one minute frequecny, and the result is confusing:
The datastore plugin debug log file show that, a call of "LoadBars" for the last M1 bars first, next it will call "LoadBars" for the last H1 bar, then some flowing call to M1 bars. and I check the log write by NewBars event, it shows the H1 time range is according to the exists M1 data's time range, and is nothing relate to data in exist 60min tables.
There is another strange thing, if I run the test on symbol "EUR/JPY", it will read the EUR/USD 60min table for once .
Posted Wednesday July 15 2009
It will attempt to load the desired frequency from the data store, which is why I mentioned having that frequency available as your minute bars are saved. The read to EUR/USD is likely a currency conversion attempt.

raidsan (7/14/2009)
It is easy to save aggregate data in datastore plugin, but the IDataStore interface could spport to work in this way?
I have use mysql plugin to run a tradingsystem under sixtyminutes frequency on symbols which use one minute frequecny, and the result is confusing:
The datastore plugin debug log file show that, a call of "LoadBars" for the last M1 bars first, next it will call "LoadBars" for the last H1 bar, then some flowing call to M1 bars. and I check the log write by NewBars event, it shows the H1 time range is according to the exists M1 data's time range, and is nothing relate to data in exist 60min tables.
There is another strange thing, if I run the test on symbol "EUR/JPY", it will read the EUR/USD 60min table for once .
Posted Friday July 17 2009
but the H1 data is already exist, RE don't call IDataAccessor.Load(sixtyminutes) to load H1 data, how can I do to let RE load exists H1 data?
Maybe I should use another symbol name e.g. H1_EURJPY,
then let the dataplugin to update H1_EURJPY bar data while the end of M1 NewBar handler .
Posted Tuesday April 26 2016
Please note that jTWSdump has moved to:
http://www.tradingsoftwarelab.com/jtwsdump/

The old link/URL is redirecting to the new location but may stop working in the future.

Edited: Wednesday January 11 2017 by wwwmbox


Similar Topics


Reading This Topic


2005-2017 © RightEdge Systems