RightEdge Forums / General / Plugin Development / Synchronizing bar timing between RE and data source / Latest Posts

RE: Synchronizing bar timing between RE and data source

dplaisted — Fri, 30 Oct 2009 10:39:52 GMT

I would recommend keeping track of the most recent tick time, and if a tick comes in with a timestamp earlier than that, modify the timestamp of that tick to match the most recent one.

Daniel

RE: Synchronizing bar timing between RE and data source

sda75 — Fri, 30 Oct 2009 03:01:35 GMT

I am working on a plugin for RE Edition2 for Russian AlfaDirect streaming data.

I have streaming data.

[28.10.2009 16:31:55] Received:

*at*

651927936|12910|740|66.17|16:31:55|341598715|0|2|0

[28.10.2009 16:31:55] Received:

*at*

111348438|27230|10|135830|16:31:54|341598715|6|2|0

111348439|27230|10|135810|16:31:55|341598715|1|2|1

111348440|27230|1|135815|16:31:55|341598715|6|2|1

[28.10.2009 16:31:56] Received:

*at*

111348442|27230|5|135820|16:31:55|341598716|6|2|0

111348443|27230|2|135810|16:31:55|341598716|1|2|1

111348445|27230|2|135805|16:31:55|341598716|1|2|1

12910 & 27230 - it's tickets from different exchanege and I can't use ticks.Sort because 651927936 (tick number) already added.

RE: Synchronizing bar timing between RE and data source

hairy_mug — Thu, 27 Aug 2009 18:57:41 GMT

Hi steve,

Aggrigation sounds like the direction but you may consider aggrigating into 10 or 15 second "slots" and then into the minute.

This way you wouldn't have all the calcs happening at the end of minute... which should work better whith large symbol sets.

If you want to put the aggration rules together, i.e. how you want the data combined, I can build you an "aggrator" class...

Walt

>> I do have interest in a TDA plug-in since they are my broker also and I have their SDK but no time (or desire to do it all)

RE: Synchronizing bar timing between RE and data source

Steve2008 — Mon, 27 Jul 2009 16:11:10 GMT

Or maybe do the aggregation in RE, in a thread.

RE: Synchronizing bar timing between RE and data source

Steve2008 — Mon, 27 Jul 2009 11:19:58 GMT

I'm facing some issues for the real plug-in. Thoughts appreciated by RE staff or anyone.

The function that gets one minute bars, TDAC_OnOHLCBar, is triggered upon receipt of a bar for one symbol. On a large symbol list such as the S&P 500 the bars come in irregularly over a period of around 5 to 20 seconds, longer sometimes. Sometimes the timestamps are old. I am not sure that bars are generated if there are no trades.

I would need to save data for each bar, and probably in a separate thread judge when bars have stopped coming in for that minute, perhaps based on receipt of data for a threshold percentage of bars plus a threshold time span of no receipt of data, and then generate ticks in the proper time sequence and send them to the listener. I might want to generate bars using last good data for symbols whose bars have not yet come in or that have bad timestamps, but not do so if the anomaly persists for more than one minute, which could indicate that the symbol has stopped trading.

The reason I'm looking at one minute bars instead of streaming data in the first place is that RE gets bogged down with tick-level data on a large symbol list.

Given the issues here, especially the lag in TDA one minute bar receipt, I'm starting to think that I would be better off aggregating streaming data into one minute bars myself in a separate process that talks to a RE plugin every minute. This will eliminate delays and give me better control, and also can potentially work with a variety of data feeds.

From what other people have posted, it seems that aggregation in a separate process is feasible from the standpoint of CPU and other resource use. Perhaps if necessary I could run several processes on different local cores or networked machines.

RE: Synchronizing bar timing between RE and data source

Steve2008 — Sun, 26 Jul 2009 23:39:48 GMT

Daniel, thanks for looking into this.

Apparently timing among symbols is more tightly coupled in Edition 2 than Edition 1.

RE: Synchronizing bar timing between RE and data source

dplaisted — Fri, 24 Jul 2009 23:09:53 GMT

You are sending ticks in the following order:

AAAA 12:00:01 Trade
AAAA 12:00:02 Trade
AAAA 12:00:03 Trade
AAAA 12:00:04 Trade
AAAA 12:01:00 Bid
BBBB 12:00:01 Trade
BBBB 12:00:02 Trade
BBBB 12:00:03 Trade
BBBB 12:00:04 Trade
BBBB 12:01:00 Bid
etc.

The problem is that the last AAAA tick is from 12:01:00. Then you send the first BBBB tick, and the time is 12:00:01. So this tick is received out of order. When RightEdge got the tick for 12:01:00, it took that as the current time, which means that the 12:00 bar would have been completed at that point for all symbols in your system. So when you get to the BBBB data it is too late to process that as part of the 12:00 bar.

Basically you want to sort the ticks by timestamp before sending them out. It may be more complicated in the real plugin, but in the code you sent us, you just need to add the following line after all the ticks have been added to the list but before calling the tickListener:

ticks.Sort(delegate(SymbolTick t1, SymbolTick t2) { return t1.tick.time.CompareTo(t2.tick.time);});

Thanks,
Daniel

RE: Synchronizing bar timing between RE and data source

Steve2008 — Wed, 22 Jul 2009 17:55:49 GMT

UPDATE: I just sent it, and then a corrected version to fix a bug.

RE: Synchronizing bar timing between RE and data source

billb — Wed, 22 Jul 2009 17:23:53 GMT

You can email it to resupport at rightedgesystems.

RE: Synchronizing bar timing between RE and data source

Steve2008 — Wed, 22 Jul 2009 16:51:29 GMT

I am able to replicate the problem with a lightly modified version of the supplied random data realtime plugin. It works properly when compiled for and installed in ED1, and produces the same type of problems noted above in ED2.

Let me know if you would like the code. I have had problems posting stuff here -- quota errors -- so I can e-mail it. I can send either the C# file or the entire VCS Express 2008 projects, zipped.

RE: Synchronizing bar timing between RE and data source

Steve2008 — Wed, 22 Jul 2009 14:04:09 GMT

UPDATE: I compiled this same plugin for Edition 1, and it is working correctly there. I changed the tick type for the new-minute tick from current time to previous close (for my purposes, meaningless and harmless), because current time is an ED2 innovation.

Wondering whether the bar type in ED2 is the issue, I changed it there, too, to previous close. That does not help.

So this is an ED2 issue.

RE: Synchronizing bar timing between RE and data source

Steve2008 — Wed, 22 Jul 2009 12:42:40 GMT

I am getting an orange error banner at the bottom of RE saying "Data from [begin datetime] to [end datetime] was dropped because it is out of order."

I sending 5 ticks every one minute bar as suggested (4 trade, OHLC, and 1 current time at beg of next minute) on a symbol list of 7 symbols, freshly created, and purged/reentered to ensure no residual previous data.

One symbol is updating properly in RE every minute. The others are not displaying any data, apparently because RE is dropping the ticks I send. All symbols are processed identically.

Diagnostic code shows that I am properly receiving data from TDA, and am putting correct values into the tick's time and symbol fields before calling tickListener. I am sending timestamps of 1, 2, 3 and 4 seconds into the even minute for the trade ticks and on the even next minute for the current time tick, and am working off my computer's clock, eliminating the possibility that the TDA timestamps are the issue.

Assistance would be appreciated. Thanks.

RE: Synchronizing bar timing between RE and data source

Steve2008 — Fri, 17 Jul 2009 22:06:35 GMT

Sounds like what I need. Thanks.

RE: Synchronizing bar timing between RE and data source

dplaisted — Fri, 17 Jul 2009 21:15:48 GMT

RightEdge bases it's time off of the ticks that come in from the data plugin, not off of your computer's internal clock. (Some data plugins such as the TWS plugin don't get time updates from the broker with every tick so the time is partly based on the computer's clock, but this is up to the plugin to decide.)

RightEdge won't generate a new bar for your system until it receives a tick that would be part of the next bar. There is now a TickType of CurrentTime that you can use. So to send a 1M bar that starts at 12:00, you would send the OHLC ticks with timestamps between 12:00:00 and 12:00:59, and then send a CurrentTime tick with a timestamp of 12:01:00 which would force the bar to be sent.

Thanks,
Daniel

RE: Synchronizing bar timing between RE and data source

Steve2008 — Fri, 17 Jul 2009 18:56:47 GMT

Hi Bill,

Please bear with me as I try to clarify how this works.

I will be receiving TDA's 1 minute bars whenever they send them, presumably every minute regardless of whether there were trades during that minute; if not, I can simulate this within my plugin, so we can assume that it will effectively happen. The record contains fields for OHLC, and my plugin would send to RE either 4 trade ticks, one for each value, or only 1 trade tick, the C value, depending on what the particular trading strategy looks at. My plugin may or may not send any non-trade ticks; let's assume that it will not.

The time I get a TDA 1 minute bar is not necessarily at a minute boundary per my computer's clock.

If I send RE one trade tick per minute, will it immediately generate a 1 minute bar with identical values for OHLC, and pass control to my trading strategy? This is what I wish to happen. Or is RE's bar creation on its own schedule, perhaps tied to my computer's clock, causing a lag of conceivably up to 59+ seconds between sending the tick to RE and RE generating the bar and invoking the trading logic?

The next question is whether there is a way I can I send 4 trade ticks in rapid succession (OHLC) each minute, and only after all 4 ticks have been sent, have RE immediately generate a 1 minute bar using those ticks, and pass control to my trading strategy.

BTW, your response seems to imply that if I send RE no ticks for a symbol within a 1 minute period, no bar will get generated for that minute for that symbol. Is this the case?

RE: Synchronizing bar timing between RE and data source

billb — Fri, 17 Jul 2009 17:50:03 GMT

RE only processes when it receives a tick, so you could send a last trade tick (which contains the same value as the 'real' last trade) to force a new bar on the first second of the new minute.

[quote][b]Steve2008 (7/17/2009)[/b][hr]I am working on a plugin for RE Edition2 for TD Ameritrade streaming data.

I have it working using the TDA ActiveX interface, but with 100 symbols RE chews up so much CPU that it becomes unresponsive, andI want to use a lot more than 100 symbols. I don't know whether the issue is with the TDA interface, RE Edition 2, or both. (It could be my code, but I don't think so.) There is a third-party TDA native C#interface that is conceivably more efficient, but months ago I wrote a RE plugin for it and found a bug that makes it unusable for multiple symbols. A new version is in the works, but it is not clear when it will be available.

I only need 1 minute bars, which are much less CPU-intensive than streaming data and will therefore probably alleviate the CPU bottleneck.TDA can deliver 1 minute bars, and I know from previous discussions here that I can feed these into RE by generating 4 ticks in my plugin: one each for OHLC of the bar. I have not yet programmed this, but it seems straightforward.

My question regards time synchronization. I would like RE to complete its1 minute bar immediately after delivery of the ticks, and immediately invoke trading strategy logic. I want to avoid a situation where the RE timing loop for bar creation/strategy invocation is not synchronized with that of my data source, causing a time lag.

What, if anything, can I do to ensure time synchronization?

Thanks.[/quote]

Synchronizing bar timing between RE and data source

Steve2008 — Fri, 17 Jul 2009 12:47:28 GMT

I am working on a plugin for RE Edition2 for TD Ameritrade streaming data.

I have it working using the TDA ActiveX interface, but with 100 symbols RE chews up so much CPU that it becomes unresponsive, and I want to use a lot more than 100 symbols. I don't know whether the issue is with the TDA interface, RE Edition 2, or both. (It could be my code, but I don't think so.) There is a third-party TDA native C# interface that is conceivably more efficient, but months ago I wrote a RE plugin for it and found a bug that makes it unusable for multiple symbols. A new version is in the works, but it is not clear when it will be available.

I only need 1 minute bars, which are much less CPU-intensive than streaming data and will therefore probably alleviate the CPU bottleneck. TDA can deliver 1 minute bars, and I know from previous discussions here that I can feed these into RE by generating 4 ticks in my plugin: one each for OHLC of the bar. I have not yet programmed this, but it seems straightforward.

My question regards time synchronization. I would like RE to complete its 1 minute bar immediately after delivery of the ticks, and immediately invoke trading strategy logic. I want to avoid a situation where the RE timing loop for bar creation/strategy invocation is not synchronized with that of my data source, causing a time lag.

What, if anything, can I do to ensure time synchronization?

Thanks.