|
Dear Twsapi experts,
Since 10.30 gateway/tws looks like reqMktData cannot be relied upon and I am often not getting the data I am asking for. And I am talking about basic bid/ask/last etc data for popular contracts with high volume of trading on Nasdaq/NYSE (like MSFT, GOOG etc) nothing exotic. It happens so often that it makes it impossible to use reqMktData in practice (up to 10% call failure rate). Yet I've been using reqMktData for 5+ years both in test and live trading, it was pretty reliable for me, no complains so far (it was slow, sure, but reliable :) ). Moreover for gateway/tws 10.19 it works as expected still today just 10.30+ drops data. Making me believe that something got broken in 10.30 and newer software.
The issue, as I will describe below seems to be related to resource usage/management of 10.30+ software, which makes me wonder what else tws is dropping once it gets somehow pushed. I could imagine that IB is loosing some data here which used to be perfect/fine it might as well drop other api call/callback (like drooping sending an order or dropping some order status data would be really bad). Thus making it more urgent to fix it.
I of course reported the issue to IB with logs clearly showing that the gateway didn't provide the data it was asked. Moreover, I shipped them this test app which I will share below which (for me) 100% repeats the issue. And IB went silent for 10+ days. I have no idea atm if they confirm the issue and busy fixing or it just ended up at the bottom of the backlog. Thus really appreciate the help of the community here. Maybe I am not alone experiencing this issue.
My ask for community is to run the test app provided (I did one in python and one in java. both languages experience the issue more strongly pointing towards IB software issue) if more can repeat the issue (and there is no issues in the my test code :) ) maybe it will help IB to rise the urgency for fixing the issue.
I am using IBAPI 1030.01. It works for connecting to any gateway/TWS versions. I have not tested it with other IBAPI SDK version.
in this api there is folder \samples\Python\Testbed extract provided python.zip in that folder (details for Java are later). Then, run
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
this will read symbols (about 1200 stocks from big, liquid companies on Nasdaq/NYSE), connect to your gateway (maybe you use a different port). will ask for REALTIME data (if you don't have subscription to US stocks + realtime addon, FROZEN, DELAYED seem to be experience the same issue). Test also will make sure that we don't ask for more that 75 contract simultaneously and run this operation 5 loops
The trick with this bug is that it is doesn't happen for a specific contract or specific order of API calls. In my experience it happens "randomly" after the gateway was used for some time. So this test app asks for data using self.reqMktData(self.nextValidOrderId, contract, "", False, False, []) and monitors callbacks if it receives the following data ["bid", "ask", "last", "volume", "last_timestamp", "high", "low"]. it waits for 20 seconds for this data to appear and if doesn't appear (this is the bug I am talking about) it declares this call as failed and prints out as TIMEOUT. Some runs will experience more issues than others. Some will have zero issues. The important bit that in my setup 5 loops of 1200 contracts always causes some failures if connected 10.30+ gateway/TWS and zero issues on 10.19. It is true for both Linux and Windows.
A fragment of output
All failed symbols are printed during the run and summarized with info received at the end. Also reqId is printed, so one can go into gateway log and search for this ID. All the searches I did in the gateway's log it was clear that gateway really didn't send the info the app is missing (i.e. it is not the test app which missed it somehow but the gateway didn't even send it)
Notes
- Program_get_bid.py is based on Program.py. One can easily compare the differences to the "official" sample. It is really minimal difference to make sure we can ship requests without overloading API. And again this code works just fine with 10.19 gateway and fails 100% on 10.30.
- Of course it could be situation there bid/ask is missing, but those are rare for such common stocks (I have filtered stocks by market cap and I think that list has only stocks with 10+B in market cap and some trading volume, precisely so we don't run into such issues in this test). And if that is the case, there is no excuse for missing last/volume/high/low/last_timestamp anyway. It is not present on the screenshot, but happens often
- java code is should be extracted to samples\Java\samples\testbed. However make a copy of that folder so you can compare Tesbed.java and EWrapperImpl.java to original files.
- I call it with following parameter (or change directly in the code) -p 4001 -m Real-Time -d 75 -l 5 -f symbols.csv
- again if you don't have subscription, -m Delayed seems to be causing same issues. but again, Real-Time is the most interesting.
- This is not the code I run in production. I just wanted something which repeats the issue 100% and I can share it with IB and community here. So it is fast ad-hoc code which I was trying to keep as close as possible to the official samples (java was AI driven conversion from python, but I was trying to figure out if it is api language related or not). Fingers crossed I didn't do mistakes here. Feel free to comment
- In my case I see that test is failing about 3-50 symbols out of 6000 calls (5 loops of 1200 contracts). In my production where much more complex loading of the gateway is happening (read account, do trades, monitor trades etc) I see about 10% of calls failing. Which makes me believe that it has maybe something to do with resource usage/API loading of the gateway.
- dropping -m 20 or lower symbols simultaneously slightly reduces chances of dropping data, again pointing to resource issue, maybe
- If I run the test connecting to a TWS instead of the gateway the symbols which I have in the watchlist are somewhat overrepresented in the list of "failed" contracts in the test.
Regards,
AJ
|
This is a pretty tall ask, ajn. You send 2,000++ lines of source code that you ask us to run. That will generally require us to read and understand the code before we can even consider running against our IBGW/TWS instances. I cannot invest that kind of effort.
If I understand your test beds correctly (and I took only a very quick peak), your code subscribes to and then quickly cancels market data subscriptions for more than 1,000 symbols in a rapid sequence. I assume that is to work around the default 100 simultaneous subscriptions limit. And I am sure the difference in behavior between IBGW/TWS 10.19 and 10.30 is real (you seem to have done a lot of work on this), but I am not convinced yet, that this is a bug or issue with IBGW/TWS. It appears as if you are using market data requests well outside of the "design envelope" and, while that worked in the past, it does not any longer. Your code is peppered with sleep statements, which is always a big red flag for me since it makes code vulnerable the changes in timing behavior between versions of IBGW/TWS.
Calling reqMktData creates a market data subscription for a real-time data feed that is intended to stay up for some time and is a pretty expensive operation (from IBGW/TWS and IBKR infrastructure point of view). Canceling the feed, while possibly less expensive, is also not trivial. My environment makes many of these subscriptions just before the start of the trading session, keeps them open and receives the data feeds for 23 hours, and disconnects them once the session is over. I receive between 300 and 1000 callbacks in average every second for the entire 23 hours with peak periods of 10,000 callbacks per second and more. But as far as I can tell, I do not experience any loss of data with TWS/IBGW 10.30. But I will take a look another look at my data to make sure.
Not really sure what I can tell you other than this. Try a loop of reqContractDetails for all contracts you need and use the complete contracts returned by IBKR instead of your less complete "home made" contract objects during your reqMktData loops. That will increase the chance that IBGW/TWS caches the information it needs and reduces the likelihood that additional costly HMDS queries need to be made for each subsequent reqMktData call.
Jürgen
|
Thanks Jürgen,
very valid points. I was really afraid that complexity of this bug will cause problems. So I didn't write code from scratch but did least possible modifications the the official sample app IB provides as a common ground (Program.py in case of python) precisely hoping that maybe it will be bit more familiar to people and will save time and effort for everyone (at least to IB developers as I sent it first to them) and the way to look at that code is to run a comparison to the original IB code to really see the difference = the actual new code which is executed. And the code I have added there is less than 200 lines, the rest is the "full" test app, which can be ignored (or at least I was hoping people who would look at it would have more trust to IB's code :) and everyone has the original app to compare. I didn't delete that IB boiler code to make the comparison obvious, if I do the app would be tiny. I can do that easily = keep only code which gets executed if that will make everyone more comfortable)
In the nutshell
reqMktData_test (set a break point here and easy walk)
calls reqMktData_test_work in a loop
all contracts are sent to reqMktData
the process is slowed down and finished by monitor_not_done so we don't have more than x data-lines used at the same time, that function monitors that all data is received and we can cancel the subscription
the results are printed using process_and_export_data
the following wrapper callbacks take the incoming data (unfortunately reqMktData triggers all those)
tickPrice
tickSize
tickGeneric
tickString
error
That will generally require us to read and understand the code before
Actually, running against a paper account is perfectly fine (this is how I run it all day). I believe running it against a paper account with no subscription will also work (need to set DELAYED data).
Calling reqMktData creates a market data subscription for a real-time data feed that is intended to stay up for some time
Totally. In my live application I have several hundreds of data lines to work with and slightly less number of contracts I am interested in trading "now". and about 1500 - 2500 "full universe". So I go through such cycle to get latest on the market and maintain the subscription for those several hundreds I am interested in for a while (and there I run into issues that some data is not showing up for 20+ seconds (which I consider never)). Of course as I wanted the "simplest" app showing the issues. As you rightly say, it is a large chunk of code already :) I didn't want to make it larger by having another 200 lines "ok here we subscribed and waiting, and 3 min latter suddenly some of those lines lack data"
Your code is peppered with sleep statements,
Thanks, I double checked (happy to double check). The code which I added is only to not have the busy waiting while waiting for data delivered by a different thread. There is one place I added a slowdown (but it default to zero) in tickPrice as I was experimenting with hypothesis "what if the problems are caused by me processing the data too long", I didn't find correlation. I should have removed that before posting the code to reduce mental strain of readers. Sorry about that (yet again, that line does nothing). But indeed there are several cases in IB original code (which is not used in this test) which relies on sleep before cancelling. The test code data path doesn't rely on IB sending data within specific time as far as I can see.
My environment makes many of these subscriptions just before the start of the trading session
Another interesting bit. Thanks for sharing. As in this test we subscribe in the middle of the trading AND in case of TWS contracts from the watchlist definitely overrepresented in the list of failed symbols. Maybe something got broken in IB here. That is very tough to debug, as you get one shot per day :)
you are using market data requests well outside of the "design envelope" and, while that worked in the past, it does not any longer.
The worrying bit that even if I do it for 5-20 data lines at a time (as a result slowly crawling). I still see the data missing. I see that it less to do with frequency of the requests
Try a loop of reqContractDetails for all contracts
Interesting, I didn't know about this possible issue. Will try. Although since we run several loops on the same "home brew" contracts, I would expected that after the first loop all will be good. But I will try. At this point I would try anything which has any chance of solving the issue before IB retires the 10.19 gateway and would force move to 10.30
|
On Wed, Dec 18, 2024 at 12:08 AM, ajn wrote:
I didn't delete that IB boiler code to make the comparison obvious, if I do
the app would be tiny. I can do that easily = keep only code which gets
executed if that will make everyone more comfortable)
This would be greatly appreciated. If you can make the test case as small and simple as humanly possible... this will help people (esp. IBKR internal developers, who need extra help because they're mostly inept) to id and possibly fix the problem.
|
I have implemented reqContractDetails and start constructing contract using conId I received.
so instead of this
Like this
It made no difference, unfortunately. still not receiving some data
|
I created a second version where I removed all code which is not absolutely necessary It is roughly 150 line of code which one easily sees comparing Program.py from samples to Program_get_bid.py. As per first post: The file is part of \samples\Python\Testbed extract provided python.zip in that folder Then, run (or even better run from debugger and step)
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
There is 3 new functions left. Main entry , runs the test in the loop 
Actual code subscribing to data. making sure the datalines limit is not exceeded. And knowing that Subscription is expensive I typically run 65% of line limit. I.e. for test accounts where I have only 100 lines I would run 65 max. So we never run into "max data lines limit reached" error.
monitor_not_done is called after each subscription and stops new subscribtions once we reach the limit or (finalize = True) simply waits until all subscriptions returned the data or Timed out.
The wrappers are easier described by comparison to Program.py the original test app. Really all wrappers do is saving incoming data
and there is unsubscribing bit which is driven by wrappers (here on example of error), if all data is present the cancel is called and sets "sentCancelMktData" which gives signal to test loop that one received all the data
That is pretty much it.
|
On Wed, Dec 18, 2024 at 08:09 PM, ajn wrote:
where I removed all code which is not absolutely necessary It is roughly 150
line of code which one easily sees comparing Program.py from samples to
Program_get_bid.py.
Pull it out and make it stand alone. Get rid of ALL SUPERFLUOUS things like colorization constants... or no one will want to help you.
|
Hi AJN.
First off: note that `Contract.primaryExch` is not the same as `Contract.Exchange`. You might be constructing invalid Contract objects.
About reqMktData: just for fun I checked my code and out and yesterday I made 54.000 calls to it, all received a callback except few, like ~50, because of the reason #2 mentioned below.
Possible issues: 1: make sure you comply with the max 50 requests per second pacing violation. I don't know if you'll receive an error message if that's your case. 2: error code 10197 "No market data during competing session": if you're pulling reqMktData and at the same time visualizing the data in TWS, you might get this error and hence not receive an eWrapper callback.
Note I'm in C# API 10.15.2.0 - no capacity to test on another version at the moment.
Hope it helps.
I created a second version where I removed all code which is not absolutely necessary It is roughly 150 line of code which one easily sees comparing Program.py from samples to Program_get_bid.py. As per first post: The file is part of \samples\Python\Testbed extract provided python.zip in that folder Then, run (or even better run from debugger and step)
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
There is 3 new functions left. Main entry , runs the test in the loop 
Actual code subscribing to data. making sure the datalines limit is not exceeded. And knowing that Subscription is expensive I typically run 65% of line limit. I.e. for test accounts where I have only 100 lines I would run 65 max. So we never run into "max data lines limit reached" error.
monitor_not_done is called after each subscription and stops new subscribtions once we reach the limit or (finalize = True) simply waits until all subscriptions returned the data or Timed out.
The wrappers are easier described by comparison to Program.py the original test app. Really all wrappers do is saving incoming data
and there is unsubscribing bit which is driven by wrappers (here on example of error), if all data is present the cancel is called and sets "sentCancelMktData" which gives signal to test loop that one received all the data
That is pretty much it.
|
Thanks Daniel,
Just to be clear the problem is not that I am not receiving the callbacks at all. The problem is that I am receiving not entire set of the data I expect for some of the contracts (always random which contracts will be missing some data). Sometime it is bid which is missing, sometimes ask, sometimes last etc. bit random misses. And everytime I check gateway log for the request I had issues with It is clear that that piece of data is not there (say I see that for req 123456 I get bid, ask, volume but not a last value etc.).
First off: note that `Contract.primaryExch` is not the same as `Contract.Exchange`. You might be constructing invalid Contract objects.
I am 99.99% it is not the contract issue, as I am getting most of data,
Moreover, today I was experimenting with unsubscribing/subscribing for data for a contract which at this point was missing data (my definition of "missing" is that after subscription I didn't receive all of ["bid", "ask", "last", "volume", "last_timestamp", "high", "low"] within 20 seconds at least once, typically one or two would be missing), and 100% of time I was receiving the data after a second subscription. So contract part is very likely fine.
Possible issues:
1: make sure you comply with the max 50 requests per second pacing violation. I don't know if you'll receive an error message if that's your case.
2: error code 10197 "No market data during competing session": if you're pulling reqMktData and at the same time visualizing the data in TWS, you might get this error and hence not receive an eWrapper callback.
Yes, I monitor error codes. no such errors. As for pacing, it used to be needed but SDK I believe build it in and there is an option setConnectOptions("+PACEAPI"), which I use for several years now on top of own pacer anyway (just ot be safe, remember first versions were glitchy). And typically pacing errors result that you don't get any data at all. which is not current case.
About reqMktData: just for fun I checked my code and out and yesterday I made 54.000 calls to it, all received a callback except few
Great stats, :) it used to be like this for me up and including gateway 10.19.
- what is your definition of "received"? are you looking for say "last" only or a bit wider scope like I mentioned above?
- What version of gateway/TWS are you using (not API) ? Gateway 10.19 and older work fine for me (and have been for 5 years), yet once I switch gateway to 10.30 or 10.31 I immediately start seeing issues.
AJ
|
In the attached zip I have the app. It is now only 250 lines including all the IB connection and argument parsing. The working function are all the same as before ( described earlier) I just deleted everything else. I do agree, as a standalone app it gets much more readable
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
As an added bonus this app can be run from anywhere, not just from python samples folder. so long your python environment has 10.30 installed
AJ
|
Hi AJ,
I connect via TWS and my TWS version is 10.30.1s, whereas the version of my CSharpAPI.dll is 10.15.2.0.
"I didn't receive all of ["bid", "ask", "last", "volume", "last_timestamp", "high", "low"]"
I think this might be the difference. I only request snapshots every few seconds.
E.g. in C#: client.reqMktData(reqId, contractObject, string.Empty, true, false, null);
As said, 50.000+ requests per day and working fine. Of course, I only get 'the last Last' snapshot, for my app that's fine.
Hope it helps, Daniel.
In the attached zip I have the app. It is now only 250 lines including all the IB connection and argument parsing. The working function are all the same as before ( described earlier) I just deleted everything else. I do agree, as a standalone app it gets much more readable
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
As an added bonus this app can be run from anywhere, not just from python samples folder. so long your python environment has 10.30 installed
AJ
-- Daniel
|
E.g. in C#: client.reqMktData(reqId, contractObject, string.Empty, true, false, null);
yes it is the same call I am making, except I am not asking for snapshot (4th argument is False) but whole set of data to be streamed to me
self.reqMktData(orderId, contract, "", False, False, [])
Last time I checked the difference between asking for a snapshot and streaming, snapshot was much more delayed (but that was 4-5 years ago, maybe things improved since). Still streaming or snapshot, api is promising the data, we pay for it. We should be getting it :)
And even in this test. last was there, In this case bid/ask was missing. Sometimes "last_timestamp" (time of last trade) is missing, sometime high/low/volume. bit random. I believe last is present most of the time
|
Just run a couple of more tests (using the app I posted, just changed False to True and commented cancel request). Snapshot works 100% reliable as Daniel experiences i.e. snapshot delivers all that is expected and no errors printed
self.reqMktData(orderId, contract, "", True, False, [])
live data (4th argument is False) oth is the case which has the problems
self.reqMktData(orderId, contract, "", False, False, [])
It is live data is bit broken in gateway 10.30+ . We are making some progress :)
|
On Wed, Dec 18, 2024 at 12:08 AM, ajn wrote:
Actually, running against a paper account is perfectly fine (this is how I run
it all day). I believe running it against a paper account with no subscription
will also work (need to set DELAYED data).
I presume you mean the problem will still exhibit itself. Can you confirm? Because I just ran your last sample program (w/ inconsequential changes for convenience; see attached) against a demo acct using DELAYED and it finished successfully.
- TWGW ver: 10.30.1s
- IBAPI ver: 1030.01
- Platform: Linux 4.4.0-260-generic #294-Ubuntu SMP Fri Sep 27 16:14:57 UTC 2024 x86_64 GNU/Linux
Output:
Using args Namespace(data_lines=65, file='symbols.csv', global_cancel=False, loops=1, market_data_type='DELAYED', port=4002)
serverVersion:187 connectionTime:b'20241219 15:00:41 EST'
NextValidId received: 1
NextValidId set : 1000009
Executing requests
LOOP 0 START
1000074 Waiting... Symbols in work : 65, done symbols: 0
1000209 Waiting... Symbols in work : 65, done symbols: 135
1000334 Waiting... Symbols in work : 65, done symbols: 260
1000459 Waiting... Symbols in work : 65, done symbols: 385
1000584 Waiting... Symbols in work : 65, done symbols: 510
1000709 Waiting... Symbols in work : 65, done symbols: 635
1000834 Waiting... Symbols in work : 65, done symbols: 760
1000959 Waiting... Symbols in work : 65, done symbols: 885
1001080 Waiting... Symbols in work : 65, done symbols: 1006
1001207 Waiting... Symbols in work : 65, done symbols: 1133
1001292 Waiting... Symbols in work : 24, done symbols: 1259
Number of not done symbols: 0
reqId symbol done bid ask last volume high low last_timestamp
DONE reqMktData_test 51.7 seconds
ALL Tests DONE
|
Thank you very much!
yes, I can confirm it is there, at least it was all day for me on that very code and right now. (yet it is funky atm, see end of post)
can you run it with several loops? It is often happening on the second loop or later. Does account you run has real time subscription? (see end of post, why maybe one really need subscription not 100% sure yet)
Using args Namespace(port=4002, global_cancel=False, file='symbols.csv', market_data_type='REALTIME', data_lines=75, loops=5)
serverVersion:187 connectionTime:b'20241219 16:56:08 EST'
1713296 Waiting... Symbols in work : 75, done symbols: 0
1714476 Waiting... Symbols in work : 75, done symbols: 1180
TIMEOUT reqId: 1714133, symbol: MU CANCEL TIMEOUT!!!!
1714504 Waiting... Symbols in work : 18, done symbols: 1264
Number of not done symbols: 1
reqId symbol done bid ask last volume high low last_timestamp
1714133 MU 0 N/A N/A 87.03 904214 91.0 84.61 2024-12-19 22:56:09
However I did notice 2 things
- I have run the test after hours and the frequency of errors is significantly less. I get 1 failure per 5 loops. whereas during trading hours I was getting maybe 5-20 per same test. So maybe it is related somehow on how IB infra is loaded? i.e. the infra is probably less loaded now (just a guess)
- I can repeat the issue for REALTIME and DELAYED options on paper trading account which HAS market data subscriptions. However, at least now, after hours trying it on the account which DOESN'T have subscription doesn't generate issues with DELAYED. Maybe it has to do with after hours or maybe DELAYED means different things for account with subscription and without. I don't know. So I can confirm the issue for account with subscription, yet after hours it is less rare one really need to run 5 loops to be sure.
|
On Thu, Dec 19, 2024 at 10:23 PM, ajn wrote:
However I did notice 2 things
I just ran your sample program again, after hours w/ realtime, data_lines=75, loops=5 and everything worked OK using IBGW 10.30.1s. IDK what to say... I may try it again tomorrow since you indicate there's more likelihood of a problem during RTH.
However, I'm also wondering if your network connection may have slightly degraded in the recent past? Is someone streaming HDTV while you test, lol? Are you sure you can't reproduce the problem w/ 10.19? I suppose it's possible that 10.30 has just enough extra network demand than 10.19 to make a difference but...
I mention this mostly because my server has an exceptionally good connection (and I'm located close to the USA NE data center). When I look at your email headers I notice you're in Sweden, so if you're using a local computer... you might have more variation in connection quality. Just a guess since the code itself looks OK and the test results look OK.
|
On Thu, Dec 19, 2024 at 10:23 PM, ajn wrote:
It is often happening on the second loop or later.
Ok, I managed to reproduce the problem.
|
Uh, my sanity is restored :)
As for your comment about the network, yes my dev setup is in EU (yet its 250Mbit up/down fiber, typically no issues here with Internet otherwise, but of course anything can happen :) and again switching back and forth between 10.19 and 10.30 all the time, where 10.19 does't have issues) However, I can reproduce it from my test site in US just as well. Test site is in AWS with pretty biffy setup on the East Coast (us-east-1, i..e supposed to be closest to IB servers, t3.2xlarge instance aws claims 5gbit network, ok it is probably less in reality but itis not likelly a bottleneck even it is "just" 500Mbit to the Internet ).
|
Rarely is lack of "last hop" bandwidth the root cause for network issues. In my experience it is pretty much always latency. That is latency from processing, end-to-end "flight time", congestion along the route, traffic shaping games played by the ISP, or packet loss and retransmission. But since you can reproduce what you are seeing with a t3.2xlarge AWS instance, I am pretty sure that network issues are not the root cause of what you are seeing. Having said that, latency (or changes of latency) could very well make your experience better or worse.
I am still not convinced that there is a "bug" here and I was wondering, whether you have tried (or could try) some runs without timeouts. At least the Python code currently gives up and cancels the subscription after 20 seconds, right? Maybe it just takes that long occasionally when you send massive numbers of subscriptions and cancellations.
There are a few reasons why that may be the case:
- There is no interface contract or service level agreement that specifies how long it will take for the first market data ticks to arrive after a successful subscription. It may just be that it sometimes takes a while. Especially since in many cases a set of historical tick (last close, open, ...) is sent before the real-time data starts.
- The API documentation for market data snapshot says, that the tickSnapshotEnd() callback "... will indicate the snapshot reception is finished. Expected to occur 11 seconds after beginning of request." It does not guarantee that any data has been sent before the end of the 11 second period.
- I checked my records for the SPY ETF how long it takes from the market data subscription to the first data arriving. This comparison is not perfect, since my client subscribes between 16:00 and 17:00 US/Central, keeps the subscriptions active for 24 hours, and only makes few subscriptions compared to your 1,000++ runs. For about 2,500 subscriptions over the last few years, max time for the arrival of the first data tick was 6.3s with an average of 510ms and stdev of 794ms. That does not include quite a few cases where the first tick did not arrive for more than a minute and was a "LastTimeStamp" callback and not a data tick.
Just curious whether the data eventually arrives if you give it enough time.
Jürgen
PS. Just as an FYI attached recent latency and packet loss measurements for the IBKR entry points as experienced by a server outside of Chicago.
On Fri, Dec 20, 2024 at 10:10 AM, ajn wrote:
Uh, my sanity is restored :)
As for your comment about the network, yes my dev setup is in EU (yet its 250Mbit up/down fiber, typically no issues here with Internet otherwise, but of course anything can happen :) and again switching back and forth between 10.19 and 10.30 all the time, where 10.19 does't have issues) However, I can reproduce it from my test site in US just as well. Test site is in AWS with pretty biffy setup on the East Coast (us-east-1, i..e supposed to be closest to IB servers, t3.2xlarge instance aws claims 5gbit network, ok it is probably less in reality but itis not likelly a bottleneck even it is "just" 500Mbit to the Internet ).
|
On Mon, Dec 23, 2024 at 10:31 PM, Jürgen Reinold wrote:
Just curious whether the data eventually arrives if you give it enough time.
Even if it did, that wouldn't explain why it arrives in time when using 10.19. If you run the test case as prescribed, and compare the results of 10.19 vs 10.30+, you'll probably see what I mean.
I suppose it could be classified as "not a bug" if IBKR wants to intentionally degrade the service they once provided. And, as a skeptic, I wouldn't put that past them. Nevertheless, from a technical user's perspective, this is at best a [massive] shortcoming IMHO.
After all, without any SLA they could just hold data back completely, or provide the same QOS for DELAYED and REALTIME. It's an absurd proposition so I'm leaning toward "this is a bug (full stop)".
Playing that much faster and looser w/ their specifications would leave a bad taste in an engineer's mouth... although a lawyer might love it.
:-/
|
buddy