Profile Picture

BarData checker for select date range

Posted By alpha23 6 Years Ago
Message
Posted Tuesday June 14 2011
Is there a way to limit the dates that the bar data checker will verify? The checker is a good tool; however, on minute data the process can be extremely long for multiple year data on numerous symbols. After the data has been initially checked, I would like to only verify the most recent downloaded data which will expedite this process.

Thanks, Duane
Posted Tuesday June 14 2011
And/or alternatively, can you provide the full list of what the data checker is checking for? Just reviewing another thread and your suggestion of creating a strategy to handle the same will also work. The full list will help expedite the coding.

Ultimately, being able to run the checker on a limited date range will be helpful.

Thanks, Duane
Posted Wednesday June 15 2011
It calls the BarUtils.IsValidBar method to check for invalid price values. It also checks to see if there are multiple bars for the same timestamp, and if the bar time is rounded to the bar frequency (ie you shouldn't have an hour bar with a bar start time of 09:35).

Thanks,
Daniel
Posted Wednesday June 15 2011
I created a bar with L > H and ran against an empty non-live Strategy with no call to base.NewBar() in SymbolScripts NewBar(). The exception below was produced. Is there a way to catch so that the warning can be output in SymbolScripts NewBar()?

Thanks, Duane

Invalid bar for A at 6/14/2011 12:00:00 AM: The low value is greater than the high value.
at RightEdge.Common.Internal.BarDataStreamer.GetNextBarEvent()
at RightEdge.Common.Internal.BarDataStreamer.GetNextItem()
at RightEdge.Shared.SystemWrapper.RunSystem(SystemData systemData, SharedSystemRunData runData, ServiceFactory brokerFactory)
at RightEdge.Shared.SystemWrapper.RunSystem(String filename, ServiceFactory brokerFactory, PluginSettings dataStoreSettings)
at RightEdge.Shared.SystemWrapper.RunSystem(String filename, ServiceFactory brokerFactory, PluginSettings dataStoreSettings)
at RightEdge.Shared.TradingModuleWrapper.Run(String filename)
at RightEdge.Shared.TradingModuleWrapper.RunSystem(SharedSystemRunData systemRunData)
at RightEdge.SystemProgress.InitAndRunSystem()
Posted Wednesday June 15 2011
In the symbol setup, check "Ignore Bar Data Errors".

Thanks,
Daniel
Posted Thursday June 16 2011
I see the check box when selecting Symbol Information for each symbol. Is there a way to set the same bool for all the symbols run with the strategy?

Thanks, Duane
Posted Thursday June 16 2011
dwebber (6/16/2011)
I see the check box when selecting Symbol Information for each symbol. Is there a way to set the same bool for all the symbols run with the strategy?

Thanks, Duane


There kind of is... you can write an optimization plugin and set the IgnoreDataValidation property of each symbol in the SystemRunSettings.Symbols list. Then you can just run the system once instead of doing an optimization.

Thanks,
Daniel
Posted Friday June 17 2011
Is this in build 35 or is it in a different list? I do not see the IgnoreDataValidation property in the SymbolSetup element of SystemRunSettings.Symbols. I’m currently using build 34.

Thanks, Duane
Posted Sunday June 19 2011
It's off the Symbol property of SymbolSetup.  You can set it to false for all symbols for an optimization run with the following code:
foreach (SymbolSetup ss in runSettings.Symbols)
{
   ss.Symbol.IgnoreDataValidation = true;
}

Thanks,
Daniel

Posted Thursday November 22 2012
I’m reviewing this implementation and notice setting IgnoreDataValidation = true, and then calling BarUtils.IsValidBar catches errors when values are invalid, e.g. Open is ‘0’, - per the prior post - but does not catch other invalid bar errors, e.g. Open is less than Low (only caught in ‘Bar Data Cleanup’).

How can I use all of the ‘Bar Data Cleanup’ functionality via a method call without throwing the exception noted in the prior post of this topic? Alternatively, what validation functionality is being checked in ‘Bar Data Cleanup’ that is not being done in IsValidBar? It is ideal to maintain all of the functionality in ‘Bar Data Cleanup’ as you may change this over time.

Checking data errors over a limited date range reduces validation time from hours to 10's of minutes. Further improvements can be accomplished via a multicore optimization plugin if a method call is available.

Additionally, downloads via IQFeed are re-populating previous deleted and/or fixed bars. How can this be prevented?

Thanks, Duane

Edited: Thursday November 22 2012 by alpha23


Similar Topics


Reading This Topic


2005-2017 © RightEdge Systems