Re: reqMktData subscription is not continuous
Good to know that after the Holidays, the liver is still functioning fine!
Mel
-------- Original message -------- From: "Andy Sanders via groups.io" <arteinvolo@...> Date: 2024-12-28 9:29 a.m. (GMT-08:00) To: twsapi@groups.io Subject: Re: [TWS API] reqMktData subscription is not continuous
Appeared to be that was some glitch with paper account. Liver data feed works fine.
|
Re: reqMktData subscription is not continuous
Appeared to be that was some glitch with paper account. Liver data feed works fine.
|
Re: Question on nextValidId function
thanks again Jürgen, everything is clearer now, complicated as you said, but clearer :)
BR, Marco
You are correct, Marco. Each clientId has its own sequence of orderIds and, since nextValidId() has a value of 1 when a clientId is used for the first time, the numeric values for orderIds for different clientIds can (and will be) be identical at times.
The technical requirement your client has to be aware of is that each placeOrder() call shall present an orderId that is higher than all orderIds the calling clientId has ever received in openOrder() or orderStatus() callbacks.
As long as a client receives only openOrder() callbacks for orders it placed itself, keeping a counter that is initialized from nextValidId() at connection time is sufficient. When a client terminates, TWS/IBGW will remember the highest orderId used in placeOrder() calls and assures that nextValidId() returns a value higher than that when it connects again later.
But there are cases where a client can receive openOrder() callbacks for orders placed by other clients (or TWS). In these cases, the client may need to bump the internal nextValidid counter accordingly. Let me quote directly from the TWS API documentation since it nicely describes this:
However if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks. For instance, if the client is set as the Master client, it will automatically receive order status and trade callbacks from orders placed from other clients. In such a case, any orderID used in placeOrder must be greater than the orderIDs returned in these status callbacks.
Alternatively if the function reqAllOpenOrders is used by a client, subsequent orders placed by that client must have order IDs greater than the order IDs of all orders returned because of that function call.
The second part of that statement is what you need to be aware of. Say your client needs to be know about orders for the instrument is manages that are open when it connects, a call to reqAllOpenOrders() might generate openOrder() callbacks with orderIds that are higher than the nextValidId(). In these cases, simply bump the internal counter to that orderId + 1.
And to make this very simple and safe for all scenarios, just perform that test unconditionally for all openOrder() and orderStatus() callbacks regardless of why your client receives them.
Sounds complicated, but is really not.
There is also an API Global Configuration option called "Use negative numbers to bind automatic orders." Take a look at the tool-tip.
Jürgen
On Fri, Dec 27, 2024 at 09:36 AM, Marco wrote:
Hello Jürgen,
thanks for your detailed reply. Yes I confirm my counter(s) get initialized with the value from the automatic nextValidId() callback. It is not completely clear to me when you mention "in contact". Does it mean in case different clients can manage same order?
if the clients are independently managing different orders I can have independent counters so in principle the same numerical orderId can be reused for different clients, am I correct?
thanks
Marco
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
Jürgen
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation (https://www.interactivebrokers.com/campus/ibkr-api-page/twsapi-doc/#receive-news-articles) it is stated that “if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks ”. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
|
Re: IB's New automated trading system questionnaire
I send push notifications to my mobile using an app called pushover. This works great and IB was fine with this solution.
|
Re: Question on nextValidId function
You are correct, Marco. Each clientId has its own sequence of orderIds and, since nextValidId() has a value of 1 when a clientId is used for the first time, the numeric values for orderIds for different clientIds can (and will be) be identical at times.
The technical requirement your client has to be aware of is that each placeOrder() call shall present an orderId that is higher than all orderIds the calling clientId has ever received in openOrder() or orderStatus() callbacks.
As long as a client receives only openOrder() callbacks for orders it placed itself, keeping a counter that is initialized from nextValidId() at connection time is sufficient. When a client terminates, TWS/IBGW will remember the highest orderId used in placeOrder() calls and assures that nextValidId() returns a value higher than that when it connects again later.
But there are cases where a client can receive openOrder() callbacks for orders placed by other clients (or TWS). In these cases, the client may need to bump the internal nextValidid counter accordingly. Let me quote directly from the TWS API documentation since it nicely describes this:
However if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks. For instance, if the client is set as the Master client, it will automatically receive order status and trade callbacks from orders placed from other clients. In such a case, any orderID used in placeOrder must be greater than the orderIDs returned in these status callbacks.
Alternatively if the function reqAllOpenOrders is used by a client, subsequent orders placed by that client must have order IDs greater than the order IDs of all orders returned because of that function call.
The second part of that statement is what you need to be aware of. Say your client needs to be know about orders for the instrument is manages that are open when it connects, a call to reqAllOpenOrders() might generate openOrder() callbacks with orderIds that are higher than the nextValidId(). In these cases, simply bump the internal counter to that orderId + 1.
And to make this very simple and safe for all scenarios, just perform that test unconditionally for all openOrder() and orderStatus() callbacks regardless of why your client receives them.
Sounds complicated, but is really not.
There is also an API Global Configuration option called "Use negative numbers to bind automatic orders." Take a look at the tool-tip.
Jürgen
On Fri, Dec 27, 2024 at 09:36 AM, Marco wrote:
Hello Jürgen,
thanks for your detailed reply. Yes I confirm my counter(s) get initialized with the value from the automatic nextValidId() callback. It is not completely clear to me when you mention "in contact". Does it mean in case different clients can manage same order?
if the clients are independently managing different orders I can have independent counters so in principle the same numerical orderId can be reused for different clients, am I correct?
thanks
Marco
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
Jürgen
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation (https://www.interactivebrokers.com/campus/ibkr-api-page/twsapi-doc/#receive-news-articles) it is stated that “if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks ”. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
I have executed the short file you posted multiple times through IB_Gateway.
Final result in prompt:
done symbols: 1283
Number of not done symbols: 0
In about 52 seconds.
Also, gateway is sending with each ticker
2024-12-27 13:09:32.976 [OQ] INFO [JTS-usfarmDispatcherS3-52S3-53] - Sending for ticker=1001373 market data info: bboExchangeAndSecType=a60001, snapshot perms=REALTIME_TOP
I noticed that I don't have any output files from this script. Could you send the code you used to write the bid data to file.
I am on TWS 10.30.1, did you recently update to 10.30 or have you been fighting with this for a while?
|
Re: Question on nextValidId function
Hello Jürgen, thanks for your detailed reply. Yes I confirm my counter(s) get initialized with the value from the automatic nextValidId() callback. It is not completely clear to me when you mention "in contact". Does it mean in case different clients can manage same order? if the clients are independently managing different orders I can have independent counters so in principle the same numerical orderId can be reused for different clients, am I correct? thanks Marco
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
Jürgen
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation (https://www.interactivebrokers.com/campus/ibkr-api-page/twsapi-doc/#receive-news-articles) it is stated that “if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks ”. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
|
Re: Question on nextValidId function
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
Jürgen
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation (https://www.interactivebrokers.com/campus/ibkr-api-page/twsapi-doc/#receive-news-articles) it is stated that “if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks ”. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
|
Question on nextValidId function
Hi All, one question on the orderId and the necessity of nextValidId function. From the documentation (https://www.interactivebrokers.com/campus/ibkr-api-page/twsapi-doc/#receive-news-articles) it is stated that “if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks ”. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed? thanks, Marco
|
reqExecution failed to get executions
I have a Java program using TWS API to get market data and place orders. After it made a trade, it gets "ExecDetails" and "CommissionReport" callbacks. But after I restarted the Java program and called "clientSocket.reqExecutions(reqId, new ExecutionFilter());", it only gets "ExecDetailsEnd" message without any executions that were done before the restart. I tried to set the "clientID" and "time" field to start of the today in the ExecutionFilter object but no use. Do I miss anything in the TWS setting? Could anyone shed some light?
|
Re: NQ Futures Security Definition
Sorry to bother. I figured it out. I need to use March 2025.
When I try to place an order on NQ futures, I have defined the contract the following way, but i get the following error: Error: 32 200 No security definition has been found for the request
contract = Contract()
contract.symbol = 'NQ'
contract.secType = 'FUT'
contract.exchange = 'CME'
contract.currency = 'USD'
contract.lastTradeDateOrContractMonth = "202412"
|
NQ Futures Security Definition
When I try to place an order on NQ futures, I have defined the contract the following way, but i get the following error: Error: 32 200 No security definition has been found for the request
contract = Contract()
contract.symbol = 'NQ'
contract.secType = 'FUT'
contract.exchange = 'CME'
contract.currency = 'USD'
contract.lastTradeDateOrContractMonth = "202412"
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Thu, Dec 26, 2024 at 05:36 AM, Jürgen Reinold wrote:
Requesting data for SMART instead
I thought this was an interesting observation as well. So, I changed the input list to just 800 of the most liquid stocks (see attached) and specified SMART as the exchange instead. Additionally, I changed the timeout to 45 seconds. Failure occurred in the same manner.
Of course, your theory that the semantics w.r.t. reqMktData have mysteriously changed still holds water. And, even though the professional way to distribute such a change would have been to implement reqMktData2 and optionally deprecate reqMktData (so that there were no unpleasant surprises), at least there appears to be some form of sour logic to what's going on.
However, without confirmation from IBKR there's no way to know this with absolute certainty, and that's not great. Regardless... it seems the best advice, at the present time, might be as previously suggested, to first initialize things with a snapshot.
If IBKR takes responsibility for any of this I'd absolutely love to know about it. Keep us posted @ajn.
|
Re: reqMktData subscription is not continuous
Snapshot parameter is set to false.
|
Re: reqMktData subscription is not continuous
You are requesting a snapshot, hence its expected to only receive one quote. If you want to subscribe permanently you can notify it on the reqMktData request.
Trying to subscribe to live Bid / Ask prices for ES futures using "reqMktData".
var contract = new Contract { Symbol = "ES", Exchange = "CME", ConId = 495512563, LastTradeDateOrContractMonth = "20251219" };
_client.TickPrice += o => Console.WriteLine(JsonSerializer.Serialize(o));
_client.ClientSocket.reqMktData(id, contract, string.Empty, false, false, null);
As a result, I receive one round of messages below, and then notifications stop.
{"Attribs":{"CanAutoExecute":true,"PastLimit":true,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":5801,"Data":5801,"RequestId":3,"Field":1}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":{"CanAutoExecute":true,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6300,"Data":6300,"RequestId":3,"Field":2}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":0,"Data":0,"RequestId":3,"Field":5}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6274,"Data":6274,"RequestId":3,"Field":9}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6050.5,"Data":6050.5,"RequestId":3,"Field":14}
Are there any settings, like "Return only snapshots instead of continuous subscriptions"?
Why notifications stop after receiving only one quote?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Thu, Dec 26, 2024 at 05:36 AM, Jürgen Reinold wrote:
It may very well be that IBGW/TWS 10.19, upon subscription, send some "recent
historical values" and 10.30+ are now cleaner and send "only market values
that changed".
The above is an interesting thought. But, I feel obligated to note it would be a breaking change to the interface. So basically still a pretty big "bug/problem"... just not in the implementation.
And even if it's "cleaner" over the wire, changing an interface without also changing its signature is terrible practice because it silently breaks end user applications (as @ajn may have found out).
With all due respect Jürgen, dismissing this or trying to spin it as a good thing(tm) strikes me as extremely forgiving or just naive.
Finally, if this speculation is correct, it makes me think that extra caution should be exercised prior to using 10.30+ in production. Since, I wouldn't want to have any code running the relied on defaults being set to "recent historical values" and didn't immediately fail loudly when such an assumption quietly disappeared; which seems like a possibility.
|
reqMktData subscription is not continuous
Trying to subscribe to live Bid / Ask prices for ES futures using "reqMktData".
var contract = new Contract { Symbol = "ES", Exchange = "CME", ConId = 495512563, LastTradeDateOrContractMonth = "20251219" };
_client.TickPrice += o => Console.WriteLine(JsonSerializer.Serialize(o));
_client.ClientSocket.reqMktData(id, contract, string.Empty, false, false, null);
As a result, I receive one round of messages below, and then notifications stop.
{"Attribs":{"CanAutoExecute":true,"PastLimit":true,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":5801,"Data":5801,"RequestId":3,"Field":1}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":{"CanAutoExecute":true,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6300,"Data":6300,"RequestId":3,"Field":2}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":0,"Data":0,"RequestId":3,"Field":5}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6274,"Data":6274,"RequestId":3,"Field":9}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6050.5,"Data":6050.5,"RequestId":3,"Field":14}
Are there any settings, like "Return only snapshots instead of continuous subscriptions"?
Why notifications stop after receiving only one quote?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
I still believe there is no bug or anything wrong with reqMktData real time feeds. And snapshots are probably what you are looking for in the first place.
TWS API documentation says about reqMktData snapshots that: ... it is possible to request a snapshot of the current state of the market once instead of requesting a stream of updates continuously as market values change.
What you experience is likely that, at times, not all of the market values you are interested in change during the 20 seconds you are looking at the market. You could easily verify that yourselves by downloading TickByTick historical data for the time periods that fail your test.
I took a quick look at the "missing" BID price for MTDR on 20241217:
- You request real-time data for MTDR@NYSE only, instead of MTDR@SMART. On that day, NYSE handled only 22% of all trades. They handled 51% of the total volume, but I bet most of it was related to the MOO and MOC auctions.
- And then there were only 563 trades during the entire trading session (RTH and outside of RTH) on NYSE. That means there were definitely long periods of time that day (way longer than 20 seconds) where the BID price (or other "market values") just did not change and, therefore, no reports would have taken place in the real-time stream. But that would be no bug, just "nothing to report".
- Requesting data for SMART instead of only for NYSE may increase your chance of seeing "market values change" while you look. But even then, MTDR had only 2,563 trades (compared with 177,164 for APPL) and long quiet periods would have to be expected.
I don't have enough data about your 10.19 vs 10.30+ comparisons to explain the differences your are seeing. It may very well be that IBGW/TWS 10.19, upon subscription, send some "recent historical values" and 10.30+ are now cleaner and send "only market values that changed". But that is just a guess and there may be other explanations.
Jürgen
On Thu, Dec 19, 2024 at 11:56 AM, ajn wrote:
Just run a couple of more tests (using the app I posted, just changed False to True and commented cancel request). Snapshot works 100% reliable as Daniel experiences i.e. snapshot delivers all that is expected and no errors printed
self.reqMktData(orderId, contract, "", True, False, [])
live data (4th argument is False) oth is the case which has the problems
self.reqMktData(orderId, contract, "", False, False, [])
It is live data is bit broken in gateway 10.30+ . We are making some progress :)
|
Re: Entry that triggers OCO with TP and SL
переключити цитоване повідомлення
Показати цитований текст
This is not correct: all three orders must have unique order ids. But the TP and SL orders must have “ParentId” set to the “OrderId” of the main order. Create 3 separate orders. TP and SL orders will have property "OrderId" that should be the same as the main order ID. The first two orders will have property "Transmit" set to false, the last one is true.
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Mon, Dec 23, 2024 at 10:31 PM, Jürgen Reinold wrote:
Just curious whether the data eventually arrives if you give it enough time.
Twenty seconds seems to be a pretty long time (see attached).
How long do you think we should wait before crossing over from thinking this is a "feature" to "bug"?
Ah... good old Sorites paradox, never fails to amuse :-)
|
Despair
buddy
