Repository navigation
[Bug][Priority] RC sometimes not sending zero timer payloads? #1158
Description
I noticed that some sessions on the brazil bot were getting no response at times from the DF.agent.
https://ba.chatbot.vega.viasat.com/live/sBkcCy9aeq3ujxdpq/room-info
https://ba.chatbot.vega.viasat.com/live/QBdfwBdq657aza5mC/room-info
https://ba.chatbot.vega.viasat.com/live/gtP8QotNBXLEf746e/room-info
Molly suggested that DF was expecting a zero timer payload which never arrived from RC.
hey, so i am digging into the first session you sent, and https://ba.chatbot.vega.viasat.com/live/sBkcCy9aeq3ujxdpq is getting stuck on a particular page in CX that expects a RC zero timer event - but RC didn’t send the event
8:35
not sure why as that flow has been workign fine for me every time i’ve tested
8:37
is there any way to see why RC is not sending that event? it seems like a weird intermittent issue?
8:38
@Shailesh351 I will assign to you...... maybe there is something in the RC logs?
Metadata
Metadata
Assignees
Labels
No labels
Activity
ear-dev commentedon Jun 16, 2022
NOTE: Molly will provide Shailesh access to the CX logs for debugging. There are a few rooms with this issue pasted above, but I also use this one:
Fnz8PEtSYcMR5Zc6Y@Shailesh351 already has access to the CX agent so he can look at how the
boletointent andboletoPauseevents are handled. We have other zero timer payloads like registration that do not ever show this behavior where the room gets stuck because CX is not getting the pause event.One diff might be that the other payloads also include a "please wait" text message, but we currently do not think that RC handles that any differently.
Another possibility might be that RC is somehow not handling an HTTP error message in this case? Basically from the RC side, we are not seeing any errors showing up in our logs around this event.
[-][Bug] RC sometimes not sending zero timer payloads?[/-][+][Bug][Priority] RC sometimes not sending zero timer payloads?[/+]ear-dev commentedon Jun 23, 2022
@Shailesh351 will find succesfull sessions where the visitor sent 'boleto' text, and compare with our frozen sessions. We may need log points to figure this out.
Molly can provide list of sessions with the issue...... NOTE: Debug logging was on June 21-22. So sessions have to be from then.
ear-dev commentedon Jul 1, 2022
NOTE: After our recent upgrade in prod to RC version v4.4.2.widechat-4 and DF version v1.2.3.widechat-6 this issue seems to have fixed itself.... we will leave this story under review for another week and close if we do not see a repro.
ear-dev commentedon Jul 6, 2022
@Shailesh351 @bhardwajaditya looks like this issue may have returned in prod. This time it was 'verificationPause', where CX was waiting for the event. The event is eventually showing up, but 40 minutes late or so. Somehow our task scheduling is getting stuck?
ear-dev commentedon Jul 7, 2022
General Notes:
Who's responsible for the event scheduling: DF.app -> writes to DB -> appBridge -> RC.server
appIDis scheduling an onetime jobprocessor ID", "The AppappIdis scheduling a recurring jobprocessor ID"What is the lifecycle of a scheduled event?? RC docs? @bhardwajaditya can you help me document all the different states that a scheduled event will go through.
ear-dev commentedon Jul 7, 2022
@bhardwajaditya searching for the log point where a job is getting scheduled does not help because there is no identifying data associated with it. I think we should make a story to add the roomID to these log points. What do you think?
ear-dev commentedon Jul 14, 2022
NOTE: we see three flavors of this bug
@Shailesh351 can you please look at latest RC server upstream to see if they may have a fix that we're missing? Thanks.
ear-dev commentedon Jul 26, 2022
@Shailesh351 I've been testing this build and it is currently failing our 'Multiple "continue_blackout" message dropping payloads in a row ' test, described in this wiki: https://wiki.viasat.com/pages/viewpage.action?pageId=549170025
Can you verify please? thanks...