Profile Picture

RightEdge 2008 Edition 2 Beta

Posted By dplaisted 9 Years Ago
Message
Posted Tuesday November 04 2008
A beta of RightEdge 2008 Edition 2 is now available for "early adopters".  There are still rough edges, but this version does support multiple frequency systems, decoupling the data frequency from the system frequency, and better support for large data sets.

The beta is available at http://www.rightedgesystems.com/downloads/RightEdge2008-Ed2-Beta-6.exe.  Note that since this is an early beta, you may run into unexpected problems.  We do not recommend using this beta for live trading.

See the following post for information on what has changed and how to use the new features.

Thanks,
Daniel

Posted Tuesday November 04 2008
Guide to RightEdge 2008 Edition 2
 
This post will introduce the new features in RightEdge 2008 Edition 2.  This version is still under development, so things will change before the final release.
 
Breaking changes, things that are broken, and other important information
  • The documentation installed with RightEdge has not been updated yet.
  • You will probably need to copy your license file to the installation directory for this edition.
  • Project files from RightEdge 2008 Edition 2 cannot be opened in previous versions of RightEdge.  When you open a project file from an older version of RightEdge, a copy of that project file will be saved with the previous version number added to the filename so that you can still open it up in the older version of RightEdge.  For example, if the project filename is MySystem.rep, a copy named MySystem.2008.1.0.0.rep will be saved.
  • If you use a custom data storage plugin, you will not have a data storage plugin set up in RE 2008 Edition 2.  You will need to go to the options dialog and choose your plugin there.
  • Progress bar doesn't show progress in system progress dialog.
  • RList class
    • No longer has AsIList() method or InnerList property.
    • Count now refers to the number of available items.  This may be less than the total number of bars if MaxLookBack is not zero.  The TotalBars property will get the total number of bars.
  • System results plugins no longer have direct access to bar history.  Some of the built-in system results are not calculated (they should display as 0 or n/a).
  • The SystemData.TradeInsideBars property has been removed.  There is now a CreateTicksFromBars property (which is set to true by default) and an EnableTradeOnClose (which is set to false by default).
  • The BarOpened event has been removed.
 
 
Bar/Tick Generation
The frequency of your system is no longer tied to the frequency of the  input data.  The main system frequency is specified in the trading system options.  This controls the frequency for many things such as when your NewBar methods will be called, the default frequency for your indicators, the frequency that system statistics will be saved, etc.  Right now you are limited to a set of predefined frequencies or "Tick" for the system frequency.  If set to "Tick", then the frequency of the symbols you have selected in the watchlist when you run the system will be used for the system frequency.  Otherwise, the input data will be aggregated to the specified frequency and passed to your system.

The input data bars for your system are sent to a tick generator.  If the CreateTicksFromBars property in the SystemData class is set to true, the tick generator breaks the bars down into ticks.  Currently the tick generator just generates 4 ticks for each bar, for the open, low, high, and close.  We will probably allow you to supply your own tick generator, which would allow you to determine whether the low or high tick comes first, or to randomly generate more ticks in between these values.

The ticks from the tick generator are sent to the frequency manager for the system.  This aggregates the ticks into the system frequency, and raises the NewTick and NewBar events.  The paper broker fills orders based on ticks, which means you can use higher frequency data on a lower frequency system for a more accurate simulation.  For example, it is possible that both the profit target and stop loss for a position could be triggered within the range of a daily bar.  If you use higher frequency data (say 15 minute bar data), then it is more likely that the simulation will reflect which price would have been hit first. 
 
 
Multiple Frequencies
RightEdge now has support for multiple frequencies.  The Frequency class represents a given frequency for a given symbol.  It provides a list of bars for the frequency, as well as events for NewBar and NewTick for that frequency.  There are GetFrequency methods on the symbol script and SystemData classes which allow you to retrieve a Frequency object for the desired frequency and symbol.  RightEdge will only start building bars for the frequency after you have requested it by calling GetFrequency, so generally you will want to call it in your system startup code.  To set an indicator to a different frequency, call SystemData.IndicatorManager.SetFrequency().  If the indicator needs a series as input, and you want to use a bar element series for that input, be sure to use the bar element series from the corresponding Frequency class.

The code below shows how to subscribe to bars in a different frequency and how to create an indicator using that frequency.

public class SampleSymbolScript : MySymbolScriptBase

{

      Frequency HourlyFreq;

      SMA sma25;

      SMA sma5Hourly;

     

      public override void Startup()

      {

            //    Create indicator using system frequency

            sma25 = new SMA(25, Close);

 

            //    Get hourly frequency and subscribe to new bars

            HourlyFreq = GetFrequency(BarFrequency.SixtyMinute);

            HourlyFreq.NewBar += NewHourlyBar;

           

            //    Create indicator using hourly frequency

            sma5Hourly = new SMA(5, HourlyFreq.Close);

            SystemData.IndicatorManager.SetFrequency(sma5Hourly, HourlyFreq);

      }

     

      public override void NewBar()

      {

            OutputMessage("New system bar: " + Bars.Current.PriceDateTime.ToString());

      }

     

      public void NewHourlyBar(object sender, SingleBarEventArgs args)

      {

            OutputMessage("New hourly bar: " + args.Bar.PriceDateTime.ToString());

      }

}


 
Large data set support
Most systems only need a certain amount of bar history for their calculations, and there is no need to keep bar data available past that amount.  When working with large amounts of high frequency data, it may not be possible to store all of the data in memory at once.  To allow systems to be run with large amounts of data, RightEdge now includes a MaxLookBack property in the RList class.  If its value is greater than zero, RightEdge will discard any items past the maximum lookback.  If you try to access any of the discarded items, an exception will be thrown.  You can set the value in your symbol script startup method, like this:

    Bars.MaxLookBack = 1000;

If you are using multiple frequencies, you will need to set the MaxLookBack on the bar list for each additional frequency, also.

Depending on your system, using a maximum lookback for the bar lists may or may not have a significant effect on your memory consumption.  The system statistics, indicators, and trade history, as well as possibly your system code itself, may all use a lot of memory.  If you need to decrease memory usage, you can selectively disable elements of your system and measure the effect on your system's memory usage.  You can use the following code in your system class to help with this:

      long _initialMemory;

     

      public override void Startup()

      {

            _initialMemory = GC.GetTotalMemory(true);

      }

 

      public override void Shutdown()

      {

            long finalMemory = GC.GetTotalMemory(true);

            long diff = finalMemory - _initialMemory;

 

            string msg = string.Format("Memory used: {0:n}", diff);

           

            SystemData.Output.Add(OutputSeverityLevel.Informational,

                  msg, null, "", SystemOutputLocation.OutputWindowAndStatusBar);

      }


There are four sets of system statistics stored (long, short, all, and buy and hold).  They save a snapshot is saved for each system bar.  If the system is only running with a single symbol, the system statistics memory usage will be significantly larger than the memory used by the symbol's bar data.  However, the system statistics memory usage doesn't increase with the number of symbols.  So if you are using many symbols the system statistics may not affect your memory usage too much.  You can disable some or all of the system statistics by setting the Enabled property on them to false in your system's startup method:

      SystemData.SystemHistory.LongStatistics.Enabled = false;

      SystemData.SystemHistory.ShortStatistics.Enabled = false;

      SystemData.SystemHistory.SystemStatistics.Enabled = false;

      SystemData.SystemHistory.BuyAndHoldStatistics.Enabled = false;


However, with all of them disabled, the system results summary won't show much useful information.  Another way to reduce the system statistics memory usage is to reduce the main system frequency.  If your system operates on one minute bars, you could set the system frequency to daily, and use the multiple frequency support to run most or all of your system logic on the one minute frequency.

Indicators may also use a significant amount of memory.  They don't currently support a MaxLookBack setting, so they will keep all the data for all the bars of your system.  If you need to reduce memory consumption you may have to re-implement the calculations outside of a RightEdge indicator.  Each trade that your system makes also consumes some memory.  If your system trades at a very high frequency then this could lead to high memory usage.


    New plugin system
    This edition of RightEdge introduces a new plugin system.  One of the advantages of the new system is that it is easier for plugin writers to define custom settings for plugins.  A property grid is used to display the plugin settings, and XML serialization is used to save them.  This means that to add a custom setting to your plugin, you simply need to define a public property on your plugin class.  The property grid and XML serialization are standard features of .NET, and offer a high degree of flexibility in how the settings are displayed in the property grid or saved to XML.  The code below shows how a plugin can define a basic custom property.

    [DisplayName("Time-based Bar Frequency")]

    [Serializable]

    public sealed class TimeFrequency : FrequencyPlugin

    {

          [Description("Indicates the timespan or period for each bar.")]

          [DisplayName("Bar Length")]

          public TimeSpan BarLength { get; set; }

     

          public TimeFrequency()

          {

                BarLength = TimeSpan.FromMinutes(1);

          }

     

          // FrequencyPlugin methods go here

    }


    For more advanced functionality pertaining to property grid customization or XML serialization, please refer to the .NET documentation or other .NET resources.

    Currently, only data storage plugins and custom frequency plugins use the new plugin system.  The new interface for data storage plugins to use is IDataStore.  Existing plugins which use the old IBarDataStorage interface will continue to work.


    Custom frequency plugins
    RightEdge uses frequency plugins to handle bar generation.  It includes plugins for time based, tick based, and volume based bars.  Additional plugins can be created to create other bar types.  To create a frequency plugin, derive a class from the abstract FrequencyPlugin class.  This class should use public properties to define the settings for the frequency.  The actual logic for generating bars should go in a class which implements IFrequencyGenerator and created and returned by the CreateFrequencyGenerator method of the FrequencyPlugin.

    To use a custom frequency in a system, create an instance of the frequency plugin class and pass it to the GetFrequency method to get a Frequency object.  If GetFrequency is called multiple times for the same settings for the frequency plugin and the same symbol, it will return the same Frequency object.  For this to work even if different instances of the frequency plugin with the same values for the settings were used, it needs to be able to compare them to see if they are equivalent.  This is why the FrequencyPlugin class includes abstract Equals and GetHashCode methods which you must implement.
    Currently, frequency plugins besides the time-based frequency can only be used from within a system.  The main system frequency, charts, and bar data storage all only use time based frequencies.


    Miscellaneous
    • Partial bars are now stored in the PartialItem property of the RList<BarData> class.  This means that the Current property will always contain a completed bar.  There is also a HasPartialItem property which indicates whether there is a partial bar or not.
    • A bar data cleaner tool has been added.  It is accessible from the watchlist, and will find and fix problems with bars in the data store.  It will delete extra bars where there is more than one bar for a given date/time, and if the time for a bar is not rounded correctly (based on the bar frequency) it will round it.  For daily bars this means it will set the time to 12:00 midnight, for hourly bars it would round to the nearest hour, etc.
    • There are no longer separate SymbolBars and SystemBars.  Use the Synchronize Bars property of the trading system to control whether the bar lists should be synchronized.
    Posted Tuesday November 04 2008
    Some additional issues have been found with the beta:

    • Running systems with symbols where Bar Construction is not set to Trades will fail with an exception.
    • Trying to display a system chart will fail with an exception if using a higher data frequency than the system frequency.
    • The Current value of the RList class is sometimes a partial bar and sometimes not.  The CurrentIsPartial property can be used to tell whether it is.  However, we are planning to modify it so that the Current value is always a completed bar, and the partial value will be accessible with another property.
    Posted Wednesday November 12 2008
    Will this edition support backtesting on a tick basis (with a last trade, bid/ask support)?

    Regards,
    P.
    Posted Thursday November 13 2008
    I have not had a look but with

    For daily bars this means it will set the time to 12:00 midnight, for hourly bars it would round to the nearest hour, etc.

    How would this affect a FX daily bar that starts at 4am?




    Posted Thursday November 13 2008
    Purri (11/12/2008)
    Will this edition support backtesting on a tick basis (with a last trade, bid/ask support)?

    Regards,
    P.

    We do plan to support this in RE 2008 Edition 2, although the current beta does not support it.

    Thanks,
    Daniel

    Posted Thursday November 13 2008
    kaizen (11/13/2008)
    I have not had a look but with

    For daily bars this means it will set the time to 12:00 midnight, for hourly bars it would round to the nearest hour, etc.

    How would this affect a FX daily bar that starts at 4am?

    It would set the timestamp for the bar to midnight.  So you might not want to use the cleanup tool on forex data.

    Thanks,
    Daniel

    Posted Sunday December 14 2008
    A new build of the beta is available at http://www.rightedgesystems.com/downloads/RightEdge2008-Ed2-Beta-3.exe.

    Bug fixes:

    • RTH only option in TWS plugin now also applies to orders submitted
    • Bid and ask values are now correctly saved in the bar data editor
    • Orders passed to broker's SetAccountState method will now have position ID fields set correctly
    • Fixed exception that would be thrown when using DataStartDate
    • Fixed exception when displaying a system chart if using a higher data frequency than the system frequency.
    • Symbols with bar construction set to bid, ask, or mid should now work for simulation
    • SystemData.BarClosing event now works, SystemData.BarOpened has been removed

    Other changes:

    • RList partial item handling changed.  CurrentIsPartial and SetPartialItem removed, PartialItem and HasPartialItem properties added
    • Frequency interfaces have changed so that an instance of IFrequencyGenerator only handles a single symbol.
    • ISimBroker interface updated to use NewBarEventArgs instead of NewBarInfo
    • CreateTicksFromBars and EnableTradeOnClose properties replace TradeInsideBars
    Posted Monday January 19 2009
    I noticed that Edition 2 installed without an IQFeed dll. Are there plans to support IQFeed in the near future?
    Posted Tuesday February 03 2009
    A new build of the beta is available at http://www.rightedgesystems.com/downloads/RightEdge2008-Ed2-Beta-6.exe

    Changes include:

  • Fixed an OutOfRangeException that could occur if the second bar for a symbol was an empty bar.
  • Updated IDataStore interface
  • Fixed bug with BarDataStreamer with symbols with no data between the data and trade start date
  • Fixed indicators being displayed incorrectly on charts when SynchronizeBars is enabled
  • Fixed a bug with monthly bar generation
  • Updated TWS plugin to use newer version of Krs library, and added ability to handle ApiCancelled and ApiPending order states
  • PaperTrader.GetOrderedOrders is now virtual, allowing the order in which orders are processed to be customized.
  • Fixed a bug with one of the constructors for the Highest indicator.
  • Symbols with an asset class of index can now be traded.
  • Fixed a bug when running a live system with saved open positions which require exchange rate information to calculate P&L.
  • IQFeed plugin is now included
  • The AmbiguousMatchException when running simulations may be fixed
  • Fixed bug which prevent SQL server data storage plugin from working.
  • Added tick data "editor", which only supports viewing so far.
  • Yahoo plugin now ignores bars with a timestamp other than midnight.  This should help avoid overlapping bar errors when running a simulation.
  • A few people reported an exception that said "Not expecting a partial bar at this point." when running live systems with the previous beta.  I believe this is fixed, please test it out and let me know if you have any problems.

    Thanks,
    Daniel



    Similar Topics


    Reading This Topic


    2005-2017 © RightEdge Systems