Profile Picture

Exceptions thrown while automating trading

Posted By Freolad 9 Years Ago
Rated 5 stars based on 1 vote.
Message
Posted Friday December 19 2008
I'm attempting to automate some strategies and ideally I'd like them running 24x7. Some are high-frequency FX so stability is important. Smile I've used a software package called FireDaemon to launch RE as a service with the appropriate command-line parameters. So at boot time, RE gets started automatically.

However, I've encountered a number of different exceptions while testing over the past week and was hoping someone could shed some more light on them because some of them prevent the trading strategy from restarting automatically.

1. I've posted in another thread, about the "Could not convert from NZD to USD for date 1/01/0001 12:00:00 AM." exception being thrown. If I manually attempt to restart the strategy, an error gets logged in the Output window and the strategy fails to start:

Could not write to output file 'c:\dir\strategy.dll' -- 'The process cannot access the file because it is being used by another process. '

This is the only strategy that is using the file/dll. The sequence of events is:

1. RE started automatically using command line args: /W:"Watchlist" /P:"C:\dir\strategy.rep" /L
2. The NZD/USD conversion exception is thrown.
3. Manually starting the live strategy results in the above error.

The only way to recover is to shutdown RE and restart (and not accept saved positions). It seems like some part of RE has a file open on the strategy.dll even when the strategy fails to start.

2. When trading with the IB TWS plugin, this exception was thrown:

An exception of type System.ArgumentException was thrown.
Unknown value for enum type Krs.Ats.IBNet.OrderStatus: ApiCancelled
at Krs.Ats.IBNet.EnumDescConverter.GetEnumValue(Type value, String description)
at Krs.Ats.IBNet.IBClient.ProcessMsg(IncomingMessage msgId)
at Krs.Ats.IBNet.IBClient.Run()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()


After clicking on the dialog, RE then crashed. But all automated trading is halted until someone realises there's a dialog window and clicks on it and then waits for RE to restart. Cool

3. The strategy was happily trading away and then this exception was thrown:

Cash was NaN
at RightEdge.Common.BaseSystemHistory.SimNewBar(NewBarInfo info)
at RightEdge.Common.SystemData.x56e784a497b3cb60(NewBarInfo x8d3f74e5f925679c)
at RightEdge.Common.Internal.SystemRunner.NewBar(Dictionary`2 bars)
at RightEdge.Shared.SystemWrapper.NewBar(Dictionary`2 bars)
at RightEdge.Shared.SystemWrapper.NewBar(Dictionary`2 bars)
at RightEdge.Shared.TradingModuleWrapper.NewBar(Dictionary`2 bars)
at RightEdge.LiveSystemwindow.x2e01f099ee7feedb(Dictionary`2 xd3a52115666f4605)


After clicking on the dialog, RE then crashed. Again, all automated trading is halted until the dialog was closed. It could be days before someone realizes something is amis.

4. I'm trying to automate a few strategies so FireDaemon is starting up a couple of instances of RE with different command line args. I occasionally get this error when the system reboots:

An exception of type System.Configuration.ConfigurationErrorsException was thrown.
An error occurred loading a configuration file: The process cannot access the file 'C:\Documents and Settings\User\Local Settings\Application Data\Yye_Software\RightEdge.exe_StrongName_tzopmoluzdyjreykcjm2jskkq1rnwoju\2008.1.0.0\user.config' because it is being used by another process. (C:\Documents and Settings\User\Local Settings\Application Data\Yye_Software\RightEdge.exe_StrongName_tzopmoluzdyjreykcjm2jskkq1rnwoju\2008.1.0.0\user.config)
at System.Configuration.ConfigurationSchemaErrors.ThrowIfErrors(Boolean ignoreLocal)
at System.Configuration.BaseConfigurationRecord.ThrowIfParseErrors(ConfigurationSchemaErrors schemaErrors)
at System.Configuration.Configuration..ctor(String locationSubPath, Type typeConfigHost, Object[] hostInitConfigurationParams)
at System.Configuration.Internal.InternalConfigConfigurationFactory.System.Configuration.Internal.IInternalConfigConfigurationFactory.Create(Type typeConfigHost, Object[] hostInitConfigurationParams)
at System.Configuration.ClientSettingsStore.ClientSettingsConfigurationHost.OpenExeConfiguration(ConfigurationUserLevel userLevel)
at System.Configuration.ClientSettingsStore.GetUserConfig(Boolean isRoaming)
at System.Configuration.ClientSettingsStore.WriteSettings(String sectionName, Boolean isRoaming, IDictionary newSettings)
at System.Configuration.LocalFileSettingsProvider.SetPropertyValues(SettingsContext context, SettingsPropertyValueCollection values)
at System.Configuration.SettingsBase.SaveCore()
at System.Configuration.SettingsBase.Save()
at System.Configuration.ApplicationSettingsBase.Save()
at RightEdge.xf266856f631ec016.OpenProject(String filename)

Inner Exception:
An exception of type System.IO.IOException was thrown.
The process cannot access the file 'C:\Documents and Settings\User\Local Settings\Application Data\Yye_Software\RightEdge.exe_StrongName_tzopmoluzdyjreykcjm2jskkq1rnwoju\2008.1.0.0\user.config' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Configuration.Internal.InternalConfigHost.StaticOpenStreamForRead(String streamName)
at System.Configuration.Internal.InternalConfigHost.System.Configuration.Internal.IInternalConfigHost.OpenStreamForRead(String streamName, Boolean assertPermissions)
at System.Configuration.Internal.DelegatingConfigHost.OpenStreamForRead(String streamName, Boolean assertPermissions)
at System.Configuration.ClientSettingsStore.ClientSettingsConfigurationHost.OpenStreamForRead(String streamName)
at System.Configuration.UpdateConfigHost.OpenStreamForRead(String streamName)
at System.Configuration.BaseConfigurationRecord.InitConfigFromFile()


The only solution is to manually restart every instance of RE that has thrown the error. Again, it could be a while before someone realizes the strategy isn't trading.

5. Thanks to the wonders of Windows Automatic updates and automatic reboots, I logged on one morning to discover the machine had rebooted. When starting up, the live strategies had all thrown this error:

An exception of type System.InvalidOperationException was thrown.
There is an error in XML document (0, 0).
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader)
at RightEdge.Common.PositionManager.LoadOpenPositions(String fileName)
at RightEdge.LiveSystemwindow.x0fa035c1818e31d7(SystemDataCreationSettings xb6b3da7953a69f26, String xe125219852864557)
at RightEdge.LiveSystemwindow.StartLiveSystem(xfb471916970b0c9e liveDataWindow, List`1 symbols, SystemDataCreationSettings settings, SystemRunInfo runInfo, RightEdgeCompiler compiler)
at RightEdge.xf266856f631ec016.x7315d87aa80c241d()
at RightEdge.xf266856f631ec016.ExecuteCommandLineParameters()

Inner Exception:
An exception of type System.Xml.XmlException was thrown.
Root element is missing.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo(String res)
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlReader.MoveToContent()
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderPortfolioXml.Read20_PortfolioXml()


It seems like the LiveOpenPositions.xml file had become corrupted. Deleting the files and restarting fixed the problem. Not sure how best to solve this. Is it possible to automatically blow it away if its corrupted? Or else, is it possible to close it and flush it once finished writing to it? ie, write and flush to new file, and then replace existing file with new file?

6. I've configured FireDaemon to automatically start up both RightEdge and IB's TWS. However, RE starts much faster than TWS. Smile As a result, RE failed to connect because TWS wasn't yet running. I have selected to automatically reconnect every 5 seconds but the TWS plugin never connected to TWS. I had left it running for hours and TWS was definitely up and running. Not sure why it didn't connect.

When I did finally login to the machine, I manually started the live feed without any problems. However, manually starting the live strategy resulted in an output message: cannot write to strategy.dll. To solve this I had to restart RE.

These are some of the problems I've encountered when trying to automate a couple of strategies with IB so any assistance, comments, tips, work-arounds would be hugely appreciated. Smile Thanks.
5 stars.
Posted Friday December 19 2008
Hi Freold,

I'm a high-frequency FX trader as well so I feel your pain when it comes to stability.

1. I have seen this before on occasion as well, but haven't narrowed down any specific cause. As you mentioned It seems that, for some trading system exceptions, RE is not fully releasing the dll's.

2. & 6. I'm not familiar with the TWS plugin.

3. Cash was NaN?! ouch! Is that worse than cash == 0.0? Tongue (sorry, couldn't resist.)

4. It sounds to me like windows is trying to startup all your RE instances in parallel and sometimes they are colliding on the config file. Can you put your startup commands into some form of batch file and add a delay timer between the startups?

5. I would guess windows couldn't properly shutdown RE and so just killed it. Personally, I have turned of automatic install of updates. I do leave notify and download on so that windows will prompt me, but I want manual control of when the install takes place.

Cheers!
Mark

Posted Sunday December 21 2008
Hopefully we can avoid most of these exceptions, but if it is important that the system be running and you plan to leave it unattended, I think you need some way to be notified when the system stops working.  You can do this by having the system periodically send a heartbeat message to a server (or just a service running on the same machine), and if the heartbeat message stops being sent, then an email message or other type of alert will be sent to you.

2. This looks like it is due to a change in the 9.6 version of IB's API.  I will need to investigate and probably update our plugin.

3.  This exception will be thrown if the cash reported by the broker or by the system statistics is NaN.  It's hard to tell what the root cause is but it might be a division by zero somewhere (perhaps the price or the exchange rate was zero).

4.  I agree with Mark, try putting a delay between starting up different RightEdge instances.

5.  Mark is also probably right about this one.

6.  The reconnect setting only applies if the plugin is successfully connected and is then disconnected.  It won't keep retrying the initial connect.  I would start TWS first, put in a delay, and then start RightEdge.

Thanks,
Daniel

Posted Tuesday December 23 2008
I was unable to reproduce the following exception:

When trading with the IB TWS plugin, this exception was thrown:

An exception of type System.ArgumentException was thrown.
Unknown value for enum type Krs.Ats.IBNet.OrderStatus: ApiCancelled
at Krs.Ats.IBNet.EnumDescConverter.GetEnumValue(Type value, String description)

Has this happened more than once?  I can add code that I think will prevent it from happening again but it would be better to be able to reproduce it so that I can be sure it's fixed.

Thanks,
Daniel

Posted Wednesday December 24 2008
Hi Mark and Daniel,

Many thanks for your responses.

dplaisted (12/21/2008)
Hopefully we can avoid most of these exceptions, but if it is important that the system be running and you plan to leave it unattended, I think you need some way to be notified when the system stops working. You can do this by having the system periodically send a heartbeat message to a server (or just a service running on the same machine), and if the heartbeat message stops being sent, then an email message or other type of alert will be sent to you.

Agreed, sounds like the way to go. I'm hoping though that we might get the point where the watchdog service very rarely sends out alerts. I hate being woken at 3am. ;-)

3. This exception will be thrown if the cash reported by the broker or by the system statistics is NaN. It's hard to tell what the root cause is but it might be a division by zero somewhere (perhaps the price or the exchange rate was zero).

I've encountered it a few more times now. I'm going to enable some logging of the incoming rates to the strategy and see if there's anything anomalous in the rate feed. Do you have any other suggestions to try and track it down?

4. I agree with Mark, try putting a delay between starting up different RightEdge instances.

Agree, re: putting in a delay at system startup and I've now done this. However, I don't think I can guarantee in the future that no combination of services will stop and/or restart at the same time. So I was kinda hoping that maybe some locking / retry mechanism could be implemented inside RightEdge to fix this? BigGrin

5. Mark is also probably right about this one.

Yep, I've definitely disabled Windows Update from automatically installing updates. Hopefully that's one source of automatic reboots eliminated. But I'm sure, like most people, I can't rule out other sources of reboots from power outages to tripping over power cables, to windows crashes, and all sorts of other reasons. And in these cases, there's a good chance that the LiveOpenPositions.xml file might again get corrupted and so the system might not restart correctly and/or lose that valuable info. So, I was hoping that the storing/syncing of this data to disk could be made more robust to survive these scenarios? BigGrin

6. The reconnect setting only applies if the plugin is successfully connected and is then disconnected. It won't keep retrying the initial connect. I would start TWS first, put in a delay, and then start RightEdge.

Ok, will put in a delay. Hopefully it'll be long enough. If not, the watchdog thread will have to catch in. BTW, is there any reason not to retry the initial connect if it fails?

dplaisted (12/23/2008)
I was unable to reproduce the following exception:

When trading with the IB TWS plugin, this exception was thrown:

An exception of type System.ArgumentException was thrown.
Unknown value for enum type Krs.Ats.IBNet.OrderStatus: ApiCancelled
at Krs.Ats.IBNet.EnumDescConverter.GetEnumValue(Type value, String description)


Has this happened more than once? I can add code that I think will prevent it from happening again but it would be better to be able to reproduce it so that I can be sure it's fixed.

Yes, I've seen it 3 times now. I'm not exactly sure how to trigger it but if I leave the system running long enough, I'll get that exception. I'm happy to act as a test guinea pig.

Is there anyway to solve the unreleased strategy dlls problem? (Point 1)

Cheers.
Posted Wednesday December 24 2008
Freolad (12/24/2008)
Is there anyway to solve the unreleased strategy dlls problem? (Point 1)

Cheers.

Let me give you a little background so that maybe you can help us determine why this is happening.  Trading system compile into DLLs and run in their own app domains.  This allows us to load and unload the trading system DLLs at our choosing.  Without all of this (and it's quite involved), this would not be possible.  So what it sounds like, an exception is thrown or some state is encountered where our code to unload the app domain, thereby, unloading/unlocking the trading system DLLs isn't being called.

Could you pinpoint this to a scenario like I'm describing?  Is it reproducible?

Posted Wednesday December 24 2008
I have a fix to try for the IB TWS plugin exception.  Are you using the beta for RE 2008 Edition 2, or are you using edition 1?

Thanks,
Daniel

Posted Sunday January 25 2009
Hi Daniel,

dplaisted (12/24/2008)
I have a fix to try for the IB TWS plugin exception. Are you using the beta for RE 2008 Edition 2, or are you using edition 1?


Sorry for not replying earlier. I'm using edition 1. I'd love to test out the fix and provide feedback.

Cheers.
Posted Tuesday January 27 2009
A new build which should hopefully fix this problem will be available soon.

Thanks,
Daniel

Posted Wednesday January 28 2009
Build 382, which should fix the ApiCancelled error, is now available.

Thanks,
Daniel



Similar Topics


Reading This Topic


2005-2017 © RightEdge Systems