RightEdge Forums / General / General Discussion / RightEdge 2008 Edition 2 Beta / Latest Posts

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Tue, 26 May 2009 14:09:14 GMT

We've been working on improving the performance of running systems. The next beta should have some significant speed improvements. We're not done yet and the actual speedup will depend on your particular system.

I did find areas where the runtime is proportional to the square of the number of symbols. So if you are running systems with large sets of symbols you will hopefully see a very large speedup. If you were planning on running a system with tens of thousands of symbols, you may want to wait until the next beta.

Thanks,
Daniel

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Thu, 21 May 2009 21:11:07 GMT

[quote]I plan to always download the lowest frequency data, like 1-min bar, and then use this data as I like, as 1-hour data, daily data[/quote]
In this case you are in luck. You can set the system frequency separately from the data frequency. Download 1 minute data and select those 1 minute symbols in the watchlist. Then in the system properties select 1 hour, 1 day, etc. as your system frequency. Your system will run with bars, indicators, charts, etc all in the frequency you specified as the system frequency, with bars created by aggregating the 1 minute bars you downloaded.

Thanks,
Daniel

RE: RightEdge 2008 Edition 2 Beta

FlipFlop — Thu, 21 May 2009 18:50:08 GMT

Thanks guys ! In addition to saying I agree with myself, I would add that the frequency bar could be an input parameter of my model, an input that I want to optimize. I plan to always download the lowest frequency data, like 1-min bar, and then use this data as I like, as 1-hour data, daily data... I find it much easier to do that, instead of downloading xxx-min bar whenever I want to play with a xxx-min bar system, downloading yyy-min bar when...
Also, when I'm fishing for an idea, I will typically change the bar resolution on the graph to identify some trading opportunities ("hmmm, on the 1-hour graph it doesn't look good, let's see what happen on the 3-hour graph").

[quote][b]mark0419 (5/21/2009)[/b][hr][quote][b]Tradix (5/21/2009)[/b][hr][quote] > or you would have an indicator on the 1M graph that stayed the same for 60 bars. [/quote]

I would find this useful and i know this functionality from my other programs like AmiBroker.
It's good for visual trade checks and confirmation when we see all the values and moves from other time frames too.[/quote]

I concur wholeheartedly. I find it extremely useful to be able to see the lower resolution indicators on a higher frequency timeframe.

Cheers,
Mark[/quote]

RE: RightEdge 2008 Edition 2 Beta

mark0419 — Thu, 21 May 2009 16:47:59 GMT

[quote][b]Tradix (5/21/2009)[/b][hr][quote] > or you would have an indicator on the 1M graph that stayed the same for 60 bars. [/quote]

I would find this useful and i know this functionality from my other programs like AmiBroker.
It's good for visual trade checks and confirmation when we see all the values and moves from other time frames too.[/quote]

I concur wholeheartedly. I find it extremely useful to be able to see the lower resolution indicators on a higher frequency timeframe.

Cheers,
Mark

RE: RightEdge 2008 Edition 2 Beta

Tradix — Thu, 21 May 2009 15:33:00 GMT

[quote] > or you would have an indicator on the 1M graph that stayed the same for 60 bars. [/quote]

I would find this useful and i know this functionality from my other programs like AmiBroker.
It's good for visual trade checks and confirmation when we see all the values and moves from other time frames too.

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Thu, 21 May 2009 11:53:23 GMT

[quote][b]FlipFlop (5/20/2009)[/b][hr]
About the Multiple Frequencies features : how can I graph the hourly Moving Average variable (called [b]sma5Hourly [/b]in the example code) ? By default the Moving Average that has the same frequency as my data is graphed (variable sma25), but not the other one.

Thanks[/quote]

This isn't currently supported. Does it really make sense to graph a 1M indicator on a 1H chart, or vice versa? I would think that the time differences are too big to make it useful. Either you would have 60 data points averaged into one point on the 1H graph, or you would have an indicator on the 1M graph that stayed the same for 60 bars.

What do you think?

Thanks,
Daniel

RE: RightEdge 2008 Edition 2 Beta

FlipFlop — Wed, 20 May 2009 05:54:44 GMT

About the Multiple Frequencies features : how can I graph the hourly Moving Average variable (called [b]sma5Hourly [/b]in the example code) ? By default the Moving Average that has the same frequency as my data is graphed (variable sma25), but not the other one.

Thanks

RE: RightEdge 2008 Edition 2 Beta

Tradix — Wed, 15 Apr 2009 15:46:51 GMT

Will the memory leak also be fixed in edition 2?

As you probably already know Edition 1 has a leak at optimizations. At least Daniel was able to reproduce it. I discussed it with him 1 month ago and attached a empty project. To reproduce simply do an optimization with > 10 markets and 10 years of EOD data or similar data.

The attached simple project is without any code and results in out of memory when doing an optimization. It's the same with every project, so i reduced all my own code parts until i came to a empty project which shows that it must be something directly in RE which doesn't free up all the used memory after each optimization run.

It would be great to have this bug fixed, so we could make optimizations. Right now out of memory happens after about 50 iterations which isn't a lot.

RE: RightEdge 2008 Edition 2 Beta

billb — Wed, 15 Apr 2009 15:30:37 GMT

My guess is that they will be on par with Ed 1. But we haven't even run through the optimizer yet.

[quote][b]Tradix (4/15/2009)[/b][hr]Is it likely that backtests are faster as in Edition 1 after your optimization or will it still rather be slower?[/quote]

RE: RightEdge 2008 Edition 2 Beta

Tradix — Wed, 15 Apr 2009 15:27:11 GMT

Is it likely that backtests are faster as in Edition 1 after your optimization or will it still rather be slower?

RE: RightEdge 2008 Edition 2 Beta

billb — Wed, 15 Apr 2009 13:33:58 GMT

Optimization of code is really the last step, so no, Ed 2 is still slower backtesting than Ed 1. It will likely improve before release.

[quote][b]Steve2008 (4/15/2009)[/b][hr]To the RE developers: I've been waiting forthe Edition 2 backtest slowness issue to be fixed before working with RE again. Has this been resolved in the current beta? If not, what is theexpected time frame for resolution?

Thanks.[/quote]

RE: RightEdge 2008 Edition 2 Beta

Steve2008 — Wed, 15 Apr 2009 12:09:40 GMT

To the RE developers: I've been waiting for the Edition 2 backtest slowness issue to be fixed before working with RE again. Has this been resolved in the current beta? If not, what is the expected time frame for resolution?

Thanks.

RE: RightEdge 2008 Edition 2 Beta

kaizen — Mon, 09 Feb 2009 20:03:35 GMT

On the speed question.

I have written a MetaStock data plugin. This removes the need to run SQL server or any other database system that just adds overhead. I found any database always lags a flat file system.

The speed gained was very noticeable when running a back test over 200 stocks for up to 10yrs.

For the RE guys: Have a look at Amibroker for real speed.

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Mon, 09 Feb 2009 12:22:57 GMT

We will profile and optimize the code before the final release. It may be that because of the new frequency system there is simply more work to do, and we won't be able to make it significantly faster. If this is the case we may enable a mode to bypass a lot of the frequency processing if the frequency of the bar data you are using matches your system frequency.

Thanks,
Daniel

RE: RightEdge 2008 Edition 2 Beta

Steve2008 — Wed, 04 Feb 2009 12:17:43 GMT

Is the backest slowness a known issue, and is a resolution likely in the final release?

RE: RightEdge 2008 Edition 2 Beta

Steve2008 — Tue, 03 Feb 2009 19:36:19 GMT

Edition 2 Beta 6 feedback after backtesting an intraday trading system on 1 minute bars of the S&P 500 component stocks:

The memory saving works well, which is very welcome.

Backtesting is drastically slower than in Edition 1, which I hope will be fixed. Speed is not affected by an assignment to Bars.MaxLookBack.

RE: RightEdge 2008 Edition 2 Beta

kuhasu — Tue, 03 Feb 2009 12:46:20 GMT

Thank you:)

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Tue, 03 Feb 2009 10:42:28 GMT

A new build of the beta is available at http://www.rightedgesystems.com/downloads/RightEdge2008-Ed2-Beta-6.exe

Changes include:

Fixed an OutOfRangeException that could occur if the second bar for a symbol was an empty bar.

Updated IDataStore interface

Fixed bug with BarDataStreamer with symbols with no data between the data and trade start date

Fixed indicators being displayed incorrectly on charts when SynchronizeBars is enabled

Fixed a bug with monthly bar generation

Updated TWS plugin to use newer version of Krs library, and added ability to handle ApiCancelled and ApiPending order states

PaperTrader.GetOrderedOrders is now virtual, allowing the order in which orders are processed to be customized.

Fixed a bug with one of the constructors for the Highest indicator.

Symbols with an asset class of index can now be traded.

Fixed a bug when running a live system with saved open positions which require exchange rate information to calculate P&L.

IQFeed plugin is now included

The AmbiguousMatchException when running simulations may be fixed

Fixed bug which prevent SQL server data storage plugin from working.

Added tick data "editor", which only supports viewing so far.

Yahoo plugin now ignores bars with a timestamp other than midnight. This should help avoid overlapping bar errors when running a simulation.

A few people reported an exception that said "Not expecting a partial bar at this point." when running live systems with the previous beta. I believe this is fixed, please test it out and let me know if you have any problems.

Thanks,
Daniel

RE: RightEdge 2008 Edition 2 Beta

Zora — Mon, 19 Jan 2009 18:19:35 GMT

I noticed that Edition 2 installed without an IQFeed dll. Are there plans to support IQFeed in the near future?

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Sun, 14 Dec 2008 03:05:53 GMT

A new build of the beta is available at http://www.rightedgesystems.com/downloads/RightEdge2008-Ed2-Beta-3.exe.

Bug fixes:

RTH only option in TWS plugin now also applies to orders submitted
Bid and ask values are now correctly saved in the bar data editor
Orders passed to broker's SetAccountState method will now have position ID fields set correctly
Fixed exception that would be thrown when using DataStartDate
Fixed exception when displaying a system chart if using a higher data frequency than the system frequency.
Symbols with bar construction set to bid, ask, or mid should now work for simulation
SystemData.BarClosing event now works, SystemData.BarOpened has been removed

Other changes:

RList partial item handling changed. CurrentIsPartial and SetPartialItem removed, PartialItem and HasPartialItem properties added
Frequency interfaces have changed so that an instance of IFrequencyGenerator only handles a single symbol.
ISimBroker interface updated to use NewBarEventArgs instead of NewBarInfo
CreateTicksFromBars and EnableTradeOnClose properties replace TradeInsideBars

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Thu, 13 Nov 2008 01:09:10 GMT

[quote][b]kaizen (11/13/2008)[/b][hr]I have not had a look but with

[i]For daily bars this means it will set the time to 12:00 midnight, for hourly bars it would round to the nearest hour, etc.[/i]

How would this affect a FX daily bar that starts at 4am?

[/quote]

It would set the timestamp for the bar to midnight. So you might not want to use the cleanup tool on forex data.

Thanks,
Daniel

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Thu, 13 Nov 2008 01:08:13 GMT

[quote][b]Purri (11/12/2008)[/b][hr]Will this edition support backtesting on a tick basis (with a last trade, bid/ask support)?

Regards,
P.
[/quote]

We do plan to support this in RE 2008 Edition 2, although the current beta does not support it.

Thanks,
Daniel

RE: RightEdge 2008 Edition 2 Beta

kaizen — Thu, 13 Nov 2008 00:28:58 GMT

I have not had a look but with

[i]For daily bars this means it will set the time to 12:00 midnight, for hourly bars it would round to the nearest hour, etc.[/i]

How would this affect a FX daily bar that starts at 4am?

RE: RightEdge 2008 Edition 2 Beta

Purri — Wed, 12 Nov 2008 22:28:34 GMT

Will this edition support backtesting on a tick basis (with a last trade, bid/ask support)?

Regards,
P.

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Tue, 04 Nov 2008 23:40:51 GMT

Some additional issues have been found with the beta:

Running systems with symbols where Bar Construction is not set to Trades will fail with an exception.
Trying to display a system chart will fail with an exception if using a higher data frequency than the system frequency.
The Current value of the RList class is sometimes a partial bar and sometimes not. The CurrentIsPartial property can be used to tell whether it is. However, we are planning to modify it so that the Current value is always a completed bar, and the partial value will be accessible with another property.

RE: RightEdge 2008 Edition 2 Beta

dplaisted — Tue, 04 Nov 2008 12:37:18 GMT

Guide to RightEdge 2008 Edition 2

This post will introduce the new features in RightEdge 2008 Edition 2. This version is still under development, so things will change before the final release.

Breaking changes, things that are broken, and other important information

The documentation installed with RightEdge has not been updated yet.
You will probably need to copy your license file to the installation directory for this edition.
Project files from RightEdge 2008 Edition 2 cannot be opened in previous versions of RightEdge. When you open a project file from an older version of RightEdge, a copy of that project file will be saved with the previous version number added to the filename so that you can still open it up in the older version of RightEdge. For example, if the project filename is MySystem.rep, a copy named MySystem.2008.1.0.0.rep will be saved.
If you use a custom data storage plugin, you will not have a data storage plugin set up in RE 2008 Edition 2. You will need to go to the options dialog and choose your plugin there.
Progress bar doesn't show progress in system progress dialog.
RList class
- No longer has AsIList() method or InnerList property.
- Count now refers to the number of available items. This may be less than the total number of bars if MaxLookBack is not zero. The TotalBars property will get the total number of bars.
System results plugins no longer have direct access to bar history. Some of the built-in system results are not calculated (they should display as 0 or n/a).
The SystemData.TradeInsideBars property has been removed. There is now a CreateTicksFromBars property (which is set to true by default) and an EnableTradeOnClose (which is set to false by default).
The BarOpened event has been removed.

Bar/Tick Generation

The frequency of your system is no longer tied to the frequency of the input data. The main system frequency is specified in the trading system options. This controls the frequency for many things such as when your NewBar methods will be called, the default frequency for your indicators, the frequency that system statistics will be saved, etc. Right now you are limited to a set of predefined frequencies or "Tick" for the system frequency. If set to "Tick", then the frequency of the symbols you have selected in the watchlist when you run the system will be used for the system frequency. Otherwise, the input data will be aggregated to the specified frequency and passed to your system.

The input data bars for your system are sent to a tick generator. If the CreateTicksFromBars property in the SystemData class is set to true, the tick generator breaks the bars down into ticks. Currently the tick generator just generates 4 ticks for each bar, for the open, low, high, and close. We will probably allow you to supply your own tick generator, which would allow you to determine whether the low or high tick comes first, or to randomly generate more ticks in between these values.

The ticks from the tick generator are sent to the frequency manager for the system. This aggregates the ticks into the system frequency, and raises the NewTick and NewBar events. The paper broker fills orders based on ticks, which means you can use higher frequency data on a lower frequency system for a more accurate simulation. For example, it is possible that both the profit target and stop loss for a position could be triggered within the range of a daily bar. If you use higher frequency data (say 15 minute bar data), then it is more likely that the simulation will reflect which price would have been hit first.

Multiple Frequencies

RightEdge now has support for multiple frequencies. The Frequency class represents a given frequency for a given symbol. It provides a list of bars for the frequency, as well as events for NewBar and NewTick for that frequency. There are GetFrequency methods on the symbol script and SystemData classes which allow you to retrieve a Frequency object for the desired frequency and symbol. RightEdge will only start building bars for the frequency after you have requested it by calling GetFrequency, so generally you will want to call it in your system startup code. To set an indicator to a different frequency, call SystemData.IndicatorManager.SetFrequency(). If the indicator needs a series as input, and you want to use a bar element series for that input, be sure to use the bar element series from the corresponding Frequency class.

The code below shows how to subscribe to bars in a different frequency and how to create an indicator using that frequency.

public class SampleSymbolScript : MySymbolScriptBase

{

Frequency HourlyFreq;

SMA sma25;

SMA sma5Hourly;

public override void Startup()

{

// Create indicator using system frequency

sma25 = new SMA(25, Close);

// Get hourly frequency and subscribe to new bars

HourlyFreq = GetFrequency(BarFrequency.SixtyMinute);

HourlyFreq.NewBar += NewHourlyBar;

// Create indicator using hourly frequency

sma5Hourly = new SMA(5, HourlyFreq.Close);

SystemData.IndicatorManager.SetFrequency(sma5Hourly, HourlyFreq);

}

public override void NewBar()

{

OutputMessage("New system bar: " + Bars.Current.PriceDateTime.ToString());

}

public void NewHourlyBar(object sender, SingleBarEventArgs args)

{

OutputMessage("New hourly bar: " + args.Bar.PriceDateTime.ToString());

}

Large data set support

Most systems only need a certain amount of bar history for their calculations, and there is no need to keep bar data available past that amount. When working with large amounts of high frequency data, it may not be possible to store all of the data in memory at once. To allow systems to be run with large amounts of data, RightEdge now includes a MaxLookBack property in the RList class. If its value is greater than zero, RightEdge will discard any items past the maximum lookback. If you try to access any of the discarded items, an exception will be thrown. You can set the value in your symbol script startup method, like this:

Bars.MaxLookBack = 1000;

If you are using multiple frequencies, you will need to set the MaxLookBack on the bar list for each additional frequency, also.

Depending on your system, using a maximum lookback for the bar lists may or may not have a significant effect on your memory consumption. The system statistics, indicators, and trade history, as well as possibly your system code itself, may all use a lot of memory. If you need to decrease memory usage, you can selectively disable elements of your system and measure the effect on your system's memory usage. You can use the following code in your system class to help with this:

long _initialMemory;

public override void Startup()

{

_initialMemory = GC.GetTotalMemory(true);

}

public override void Shutdown()

{

long finalMemory = GC.GetTotalMemory(true);

long diff = finalMemory - _initialMemory;

string msg = string.Format("Memory used: {0:n}", diff);

SystemData.Output.Add(OutputSeverityLevel.Informational,

msg, null, "", SystemOutputLocation.OutputWindowAndStatusBar);

}

There are four sets of system statistics stored (long, short, all, and buy and hold). They save a snapshot is saved for each system bar. If the system is only running with a single symbol, the system statistics memory usage will be significantly larger than the memory used by the symbol's bar data. However, the system statistics memory usage doesn't increase with the number of symbols. So if you are using many symbols the system statistics may not affect your memory usage too much. You can disable some or all of the system statistics by setting the Enabled property on them to false in your system's startup method:

SystemData.SystemHistory.LongStatistics.Enabled = false;

SystemData.SystemHistory.ShortStatistics.Enabled = false;

SystemData.SystemHistory.SystemStatistics.Enabled = false;

SystemData.SystemHistory.BuyAndHoldStatistics.Enabled = false;

However, with all of them disabled, the system results summary won't show much useful information. Another way to reduce the system statistics memory usage is to reduce the main system frequency. If your system operates on one minute bars, you could set the system frequency to daily, and use the multiple frequency support to run most or all of your system logic on the one minute frequency.

Indicators may also use a significant amount of memory. They don't currently support a MaxLookBack setting, so they will keep all the data for all the bars of your system. If you need to reduce memory consumption you may have to re-implement the calculations outside of a RightEdge indicator. Each trade that your system makes also consumes some memory. If your system trades at a very high frequency then this could lead to high memory usage.

New plugin system

This edition of RightEdge introduces a new plugin system. One of the advantages of the new system is that it is easier for plugin writers to define custom settings for plugins. A property grid is used to display the plugin settings, and XML serialization is used to save them. This means that to add a custom setting to your plugin, you simply need to define a public property on your plugin class. The property grid and XML serialization are standard features of .NET, and offer a high degree of flexibility in how the settings are displayed in the property grid or saved to XML. The code below shows how a plugin can define a basic custom property.

[DisplayName("Time-based Bar Frequency")]

[Serializable]

public sealed class TimeFrequency : FrequencyPlugin

{

[Description("Indicates the timespan or period for each bar.")]

[DisplayName("Bar Length")]

public TimeSpan BarLength { get; set; }

public TimeFrequency()

{

BarLength = TimeSpan.FromMinutes(1);

}

// FrequencyPlugin methods go here

}

For more advanced functionality pertaining to property grid customization or XML serialization, please refer to the .NET documentation or other .NET resources.

Currently, only data storage plugins and custom frequency plugins use the new plugin system. The new interface for data storage plugins to use is IDataStore. Existing plugins which use the old IBarDataStorage interface will continue to work.

Custom frequency plugins

RightEdge uses frequency plugins to handle bar generation. It includes plugins for time based, tick based, and volume based bars. Additional plugins can be created to create other bar types. To create a frequency plugin, derive a class from the abstract FrequencyPlugin class. This class should use public properties to define the settings for the frequency. The actual logic for generating bars should go in a class which implements IFrequencyGenerator and created and returned by the CreateFrequencyGenerator method of the FrequencyPlugin.

To use a custom frequency in a system, create an instance of the frequency plugin class and pass it to the GetFrequency method to get a Frequency object. If GetFrequency is called multiple times for the same settings for the frequency plugin and the same symbol, it will return the same Frequency object. For this to work even if different instances of the frequency plugin with the same values for the settings were used, it needs to be able to compare them to see if they are equivalent. This is why the FrequencyPlugin class includes abstract Equals and GetHashCode methods which you must implement.

Currently, frequency plugins besides the time-based frequency can only be used from within a system. The main system frequency, charts, and bar data storage all only use time based frequencies.

Miscellaneous

Partial bars are now stored in the PartialItem property of the RList<BarData> class. This means that the Current property will always contain a completed bar. There is also a HasPartialItem property which indicates whether there is a partial bar or not.
A bar data cleaner tool has been added. It is accessible from the watchlist, and will find and fix problems with bars in the data store. It will delete extra bars where there is more than one bar for a given date/time, and if the time for a bar is not rounded correctly (based on the bar frequency) it will round it. For daily bars this means it will set the time to 12:00 midnight, for hourly bars it would round to the nearest hour, etc.
There are no longer separate SymbolBars and SystemBars. Use the Synchronize Bars property of the trading system to control whether the bar lists should be synchronized.

RightEdge 2008 Edition 2 Beta

dplaisted — Tue, 04 Nov 2008 12:33:33 GMT

A beta of RightEdge 2008 Edition 2 is now available for "early adopters". There are still rough edges, but this version does support multiple frequency systems, decoupling the data frequency from the system frequency, and better support for large data sets.

The beta is available at http://www.rightedgesystems.com/downloads/RightEdge2008-Ed2-Beta-6.exe. Note that since this is an early beta, you may run into unexpected problems. We do not recommend using this beta for live trading.

See the following post for information on what has changed and how to use the new features.

Thanks,
Daniel