Client cannot sync with server for messages

redcomethk · February 11, 2022, 1:26am

We discovered that for some of our users, the messages may lose sync between the client and the MC server.

In our intervention, we have some messages that we would like the users to browse other parts in the app. We tried to achieve that by a show-link message and then a wait 30 message to force wait for the client to send the subsequent messages after 30 seconds. However, for some users their messages got “lost” and can never be received again. i.e. from the Results of the intervention we can see that the subsequent messages were sent out but on the client side the messages are not received (or more precisely the newer messages cannot be observed. But we cannot reproduce the same case in our testing environment…)

In addition, we have set some weekly routine such that for every week, a new micro dialog will be started. In the cases above, the new micro dialog can be started properly. The users who experienced the above case can still receive the new micro dialog and keep the conversation active.

Please help advise what should we do in order to at least “recover” the messages lost sync. Thank you very much for your help.

prabhu · February 11, 2022, 9:11am

Can we get some screenshots along with the explanation? Example: MC Designer results section, lost messages in the app. We haven’t faced an issue like this before, so I am not able to guess.

Please note that the “wait” command works as follows: The messages following the wait command are sent without any delay from the server. Then the app schedules a timer for 30 minutes and then shows the subsequent messages. (Generally we recommend “wait” command for waiting for very short periods of time.) My guess is that after the show-link was sent, the user might have closed the app (removed it from the panel), so the timer might have been lost. Can you please try this option and check if you can reproduce the error? Additionally, the ideal behaviour should not lose any messages in any case.

If waiting for 30 minutes could have caused this issue, we can use the 5 minute rule tree in MC Designer to bring a similar effect where the server would wait for 30 minutes after show-link to send subsequent messages.

Looking forward to your comments
Best
Prabhu

redcomethk · February 13, 2022, 4:54am

I think we wait for 30 seconds only. The screenshot is something like this:

After a show-link command, which handles the deep link to play the Youtube video in the app. We mainly tested our app on iOS but that “close app” issue can be a cause. Let me further investigate in that direction. Thanks.

BTW, if we change / update the dialog in this kind of case while the user has started the conversation before our change, after we re-activate the intervention, will the conversation be resumed? How can we make sure that the user be resumed in the new flow? Thank you for your help.

redcomethk · February 21, 2022, 9:11am

@prabhu we have another lost sync problem. One of our users reported that the conversation suddenly got stuck as follow:

From the screenshot we can see that the last “complete message” displayed is the “咁就好啦💪” one. The next one is shown as “typing”.
From the MC server, we found the following:

The next message was actually sent out but it seems that the app have some difficulties to display it, and thus the messages were not synced. I have advised the user to try to close and re-open the app to see if the sync can take place. This time there is no wait command and have you seen anything similar to this? How should we recover the conversation if we found this kind of issues again?

Thank you very much for your help.

prabhu · February 22, 2022, 8:57am

We haven’t faced similar issue like this. Do you use the latest version of the client app?

May I also know the server configuration? The server should have at least 2 cores, otherwise the deepstream struggles to sync with the client. if you have the timestamp of this problem, can you get us the deepstream logs to check out if that is the issue.

redcomethk · March 1, 2022, 5:48pm

The log is quite large and I will try to attach the following

TRIGGER_EVENT | EEVTmessage-update/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QEO{"status":"SENT_BY_USER","type":"PLAIN","id":"c-1645243972825","user-message":"","user-text":"好，明白😎","user-timestamp":1645243972123,"last-modified":1645243997498}
TRIGGER_EVENT | EEVTmessage-update/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QEO{"id":286,"status":"ANSWERED_BY_USER","type":"PLAIN","content":"","server-message":"今個星期會因應翻你上星期嘅睡眠記錄，俾一個新嘅睡眠處方你，咁請你今個星期按照返新嘅處方去做。\n---\n好多時候都需要啲時間去適應先至會睇到一啲改善，而影片都提及過，有陣時亦會有差咗嘅感覺，係好平常嘅事。\n---\n所以建議你嘗試多個星期，睇吓效果如何..😊","format":"plain","answer-format":{"type":"select-one","options":[["好，明白😎",""]]},"message-timestamp":1645240049214,"expects-answer":true,"can-be-cancelled":false,"last-modified":1645243997498,"sticky":false,"deactivation":false,"user-message":"","user-text":"好，明白😎","user-timestamp":1645243972123}
US | for R:messages/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE by server server
S | for R:messages/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE by server server
TRIGGER_EVENT | EEVTmessage-update/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QEO{"id":287,"status":"SENT_BY_SERVER","type":"PLAIN","content":"","server-message":"咁如果有啲乜嘢唔明，我會建議你可以睇翻上個星期嘅影片重溫一吓一啲概念同埋例子。亦都可以喺資料庫度睇翻相關嘅文字資訊。\n---\n另一方面，亦可以因應自己嘅日常作息（例如調教一個更適合你嘅起身時間），調整你嘅行動計劃，嚟符合翻自己嘅需要...","format":"plain","answer-format":{"type":"select-one","options":[["收到! 😎",""]]},"message-timestamp":1645243998338,"expects-answer":true,"can-be-cancelled":false,"last-modified":1645243998338,"sticky":false,"deactivation":false}
US | for R:messages/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE by server server
INCOMING_CONNECTION | from undefined (183.179.109.109)
AUTH_ATTEMPT | 183.179.109.109: AREQ{"client-version":1,"user":"aLAZhMhc0a67Ti1eXmRLroI5g1644935862386OS8npjRkkFUeuB2LETqNpNyry","secret":"sSJy98AztL4iuhDFECLSw2HHtF58QXcLKBP1ryfminmjt1DH8TkoITOIKBtgnpD8d6D6ownwuwsO0L5ut1wYOmuEIdXbxqHdxoO04SQyf5veopLSclarWWEZYVAbBcGL","role":"participant","intervention-pattern":"#CBT-I_f","intervention-password":"123"}
S | for R:messages/aLAZhMhc0a67Ti1eXmRLroI5g1644935862386OS8npjRkkFUeuB2LETqNpNyry by server server
US | for R:messages/aLAZhMhc0a67Ti1eXmRLroI5g1644935862386OS8npjRkkFUeuB2LETqNpNyry by server server
AUTH_SUCCESSFUL | aLAZhMhc0a67Ti1eXmRLroI5g1644935862386OS8npjRkkFUeuB2LETqNpNyry participant
S | for E:message-update/aLAZhMhc0a67Ti1eXmRLroI5g1644935862386OS8npjRkkFUeuB2LETqNpNyry by aLAZhMhc0a67Ti1eXmRLroI5g1644935862386OS8npjRkkFUeuB2LETqNpNyry participant
S | for E:dashboard-update/aLAZhMhc0a67Ti1eXmRLroI5g1644935862386OS8npjRkkFUeuB2LETqNpNyry by aLAZhMhc0a67Ti1eXmRLroI5g1644935862386OS8npjRkkFUeuB2LETqNpNyry participant
CLIENT_DISCONNECTED | aLAZhMhc0a67Ti1eXmRLroI5g1644935862386OS8npjRkkFUeuB2LETqNpNyry participant
CLIENT_DISCONNECTED | 18bYDCb33WyxzSOBoxL4kM8Mg1644842746206NDrE7NuSWU4BgdFbxLlB5nlpe participant
INCOMING_CONNECTION | from undefined (182.239.90.187)
AUTH_ATTEMPT | 182.239.90.187: AREQ{"client-version":1,"user":"zIki6j16oJTeK0D6goJDFWvmx1644846458708aPWODCYbCn5onZE7ECet63EcF","secret":"OZ4WL4BfqhDTyLY82svHAP3pOTVdwzBioTZ0ZeG9TaBAGeC2ty0MZLlqrkudzMueJ3Qlozvhu5id1iiekvvlRYqzTmFaVvCdCV0jmSoKUmgIR0yh0oeYxU6rLwDxMHx1","role":"participant","intervention-pattern":"#CBT-I_f","intervention-password":"123"}
S | for R:messages/zIki6j16oJTeK0D6goJDFWvmx1644846458708aPWODCYbCn5onZE7ECet63EcF by server server
US | for R:messages/zIki6j16oJTeK0D6goJDFWvmx1644846458708aPWODCYbCn5onZE7ECet63EcF by server server
AUTH_SUCCESSFUL | zIki6j16oJTeK0D6goJDFWvmx1644846458708aPWODCYbCn5onZE7ECet63EcF participant
S | for E:message-update/zIki6j16oJTeK0D6goJDFWvmx1644846458708aPWODCYbCn5onZE7ECet63EcF by zIki6j16oJTeK0D6goJDFWvmx1644846458708aPWODCYbCn5onZE7ECet63EcF participant
S | for E:dashboard-update/zIki6j16oJTeK0D6goJDFWvmx1644846458708aPWODCYbCn5onZE7ECet63EcF by zIki6j16oJTeK0D6goJDFWvmx1644846458708aPWODCYbCn5onZE7ECet63EcF participant
CLIENT_DISCONNECTED | zIki6j16oJTeK0D6goJDFWvmx1644846458708aPWODCYbCn5onZE7ECet63EcF participant
INCOMING_CONNECTION | from undefined (42.3.184.14)
AUTH_ATTEMPT | 42.3.184.14: AREQ{"client-version":1,"user":"tZcHGwzpeLfjQvZ6K1EnbT6uY1644851186800KV9mD0C4z4yrvPqnDvXpMDfk6","secret":"TZbSOMBVd7JyzFzdDFIx7M5Wj3eOlDm6YArEH2cXKA2eNouIWJqLE9SCUFipIO7JEmMNppTRanD2hkZ1U8smYS6XuJ77icyQptT1v5ZQTIFXeWY5uRlcFYo72mFqA0Wh","role":"participant","intervention-pattern":"#CBT-I_f","intervention-password":"123"}
S | for R:messages/tZcHGwzpeLfjQvZ6K1EnbT6uY1644851186800KV9mD0C4z4yrvPqnDvXpMDfk6 by server server
US | for R:messages/tZcHGwzpeLfjQvZ6K1EnbT6uY1644851186800KV9mD0C4z4yrvPqnDvXpMDfk6 by server server
AUTH_SUCCESSFUL | tZcHGwzpeLfjQvZ6K1EnbT6uY1644851186800KV9mD0C4z4yrvPqnDvXpMDfk6 participant
S | for E:message-update/tZcHGwzpeLfjQvZ6K1EnbT6uY1644851186800KV9mD0C4z4yrvPqnDvXpMDfk6 by tZcHGwzpeLfjQvZ6K1EnbT6uY1644851186800KV9mD0C4z4yrvPqnDvXpMDfk6 participant
S | for E:dashboard-update/tZcHGwzpeLfjQvZ6K1EnbT6uY1644851186800KV9mD0C4z4yrvPqnDvXpMDfk6 by tZcHGwzpeLfjQvZ6K1EnbT6uY1644851186800KV9mD0C4z4yrvPqnDvXpMDfk6 participant
S | for R:messages/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE by server server
TRIGGER_EVENT | EEVTmessage-update/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QEO{"status":"SENT_BY_USER","type":"PLAIN","id":"c-1645244242909","user-message":"","user-text":"收到! 😎","user-timestamp":1645244242254,"last-modified":1645244243148}
TRIGGER_EVENT | EEVTmessage-update/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QEO{"id":287,"status":"ANSWERED_BY_USER","type":"PLAIN","content":"","server-message":"咁如果有啲乜嘢唔明，我會建議你可以睇翻上個星期嘅影片重溫一吓一啲概念同埋例子。亦都可以喺資料庫度睇翻相關嘅文字資訊。\n---\n另一方面，亦可以因應自己嘅日常作息（例如調教一個更適合你嘅起身時間），調整你嘅行動計劃，嚟符合翻自己嘅需要...","format":"plain","answer-format":{"type":"select-one","options":[["收到! 😎",""]]},"message-timestamp":1645243998338,"expects-answer":true,"can-be-cancelled":false,"last-modified":1645244243148,"sticky":false,"deactivation":false,"user-message":"","user-text":"收到! 😎","user-timestamp":1645244242254}
US | for R:messages/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE by server server
S | for R:messages/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE by server server
TRIGGER_EVENT | EEVTmessage-update/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QEO{"id":288,"status":"SENT_BY_SERVER","type":"PLAIN","content":"","server-message":"咁就好喇💪🏻","format":"plain","message-timestamp":1645244244000,"expects-answer":false,"can-be-cancelled":false,"last-modified":1645244244000,"sticky":false,"deactivation":false}
US | for R:messages/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE by server server
S | for R:messages/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE by server server
TRIGGER_EVENT | EEVTmessage-update/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QEO{"id":289,"status":"SENT_BY_SERVER","type":"PLAIN","content":"","server-message":"不過上星期的確同你講解咗好多關於睡眠嘅概念，都擔心你未必一次過就明白同理解曬，\n所以我哋不如而家一齊重溫翻上個星期嘅概念啦✨","format":"plain","answer-format":{"type":"select-one","options":[["好呀唔該!",""]]},"message-timestamp":1645244244064,"expects-answer":true,"can-be-cancelled":false,"last-modified":1645244244064,"sticky":false,"deactivation":false}
US | for R:messages/nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE by server server
INCOMING_CONNECTION | from undefined (203.145.95.108)
AUTH_ATTEMPT | 203.145.95.108: AREQ{"client-version":1,"user":"AEjQuup3KCI0RFtHt4n08JNOF1645020559781yuQufNlCyHWv9SLjjzx19IJRx","secret":"ako2dRJiCP6CbPiifdwLj7CFTYT3WEN8G78oUpxCt7Pshs4QI08H5BELQWZyuRPOQ4uohiigqqsUUApQopG3kL91nSUOWlZ9TBnmSFq3wIFBBnR5Dlq1pRq13q3Bchqh","role":"participant","intervention-pattern":"#CBT-I_f","intervention-password":"123"}
S | for R:messages/AEjQuup3KCI0RFtHt4n08JNOF1645020559781yuQufNlCyHWv9SLjjzx19IJRx by server server
US | for R:messages/AEjQuup3KCI0RFtHt4n08JNOF1645020559781yuQufNlCyHWv9SLjjzx19IJRx by server server
AUTH_SUCCESSFUL | AEjQuup3KCI0RFtHt4n08JNOF1645020559781yuQufNlCyHWv9SLjjzx19IJRx participant
S | for E:message-update/AEjQuup3KCI0RFtHt4n08JNOF1645020559781yuQufNlCyHWv9SLjjzx19IJRx by AEjQuup3KCI0RFtHt4n08JNOF1645020559781yuQufNlCyHWv9SLjjzx19IJRx participant
S | for E:dashboard-update/AEjQuup3KCI0RFtHt4n08JNOF1645020559781yuQufNlCyHWv9SLjjzx19IJRx by AEjQuup3KCI0RFtHt4n08JNOF1645020559781yuQufNlCyHWv9SLjjzx19IJRx participant
CLIENT_DISCONNECTED | AEjQuup3KCI0RFtHt4n08JNOF1645020559781yuQufNlCyHWv9SLjjzx19IJRx participant
CLIENT_DISCONNECTED | nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE participant
INCOMING_CONNECTION | from undefined (112.119.69.145)
AUTH_ATTEMPT | 112.119.69.145: AREQ{"client-version":1,"user":"v65BeBwbwkeOWDBpb6VEABSTt1642427166824WYRJVwwSw7RNEGRN4SUY5aOpx","secret":"dSnbhaw5oJx7WowT3v8JkMgHPecXH43LhwkpjrPwPpkC8iZH5XiwKAh8ks1fWKmHI7wso47PDJRwBybEzmX1tyRmxSXDtqT1TjqB2zA5aX41iaZcQMjF8tr5dxhQphHQ","role":"participant","intervention-pattern":"#CBT-I_f","intervention-password":"123"}
S | for R:messages/v65BeBwbwkeOWDBpb6VEABSTt1642427166824WYRJVwwSw7RNEGRN4SUY5aOpx by server server
US | for R:messages/v65BeBwbwkeOWDBpb6VEABSTt1642427166824WYRJVwwSw7RNEGRN4SUY5aOpx by server server
AUTH_SUCCESSFUL | v65BeBwbwkeOWDBpb6VEABSTt1642427166824WYRJVwwSw7RNEGRN4SUY5aOpx participant

It seems that the id of the user involved is nbtHEhThrRvivyWEYggcilhIR1642157830450m3O2mqttFvlDxIqsHLTxYJ4QE. We can see the last TRIGGER_EVENT was the last message I sent in previous screenshots. And then I can only saw the user got disconnected… After some time the auth attempt is success but after some time she will get disconnected again.

redcomethk · March 1, 2022, 5:52pm

For the CPU issue, we use 4 vCPU on GCP, which should be similar to a server with 2 physical cores. Anyway from the suggestion from GCP the utilisation is low (~20%) and I think it should not be the problem?

prabhu · March 3, 2022, 10:06am

So, I think with the given information, CPU is not the problem. The user getting “disconnected” could be because of a lot of reasons. Closing the app (by swiping up) should also disconnect the app. But according to the screenshot, I see the tick which means that the connection is on.

Is the client built from a recent version in our repo? I think maybe the app is hanging.

Were you able to replicate it on your side? Please do send me the deepstream logs to psanthanam@ethz.ch.

redcomethk · March 21, 2022, 7:34am

I have another similar case from another user… The user reported that the coach was always “typing” the input

But from the MC server’s logging, the message was sent out and waiting for answer.

I think the user has closed the app and reopen but seems the connection cannot be resumed.

BTW, as the original cert expired we have updated the cert this weekend. I think I can send the most recent deepstream logs to you and also the one we collected before. I guess the issue is not an individual one so please help check and see if you find and clues. Thanks.