Rocket League event regression

Issue Description:

Based on our latest testing, event capturing has become significantly less reliable. However, our app hasn’t changed. My suspicion is that this is due to the RL event breakage on the 27th and that the fixes have regressed in parity with the old events. The two issues we’ve observed are as follows:

  1. Sometimes no events come through to OLP from OW at all
  2. Sometimes events come in as normal, but we no longer get rosterChange at the start of the match

In the case of issue 1, it’s very difficult to provide much, since the app works as normal until we get into RL and then we just hear nothing from the OW event API.

In the case of issue 2, we used to always get rosterChange at the start of a match. Now we don’t.

Steps to reproduce:

Just play an RL match using OLP, the same way you have been. After the match has been played, you’ll likely see that OLP doesn’t show any results. When either of these steps happen, the outcome is the same: OLP is waiting to get valid results from both players, so the challenge never finishes.

Impact for my app: [low, mid, high, show-stopper]

This is a show-stopper for OLP’s launch.

Please attach a zip package of your logs

Below are zips for both issues.

OverwolfLogs_2019-09-02_14-30-32-no-roster-change.zip (1.1 MB) OverwolfLogs_2019-09-02_14-20-02-no-events.zip (1.0 MB)

we will check it and let you know ASAP. thanks.
@Shargaas FYI.

@jeaye, @eransharv

Hey guys,

We managed to re-create the scenario where “rosterChange” is not working and we opened a bug on it for our devs. Hopefully we’ll get to fix that for you guys in the up coming weeks. Can’t give an exact estimate at this point (we’re in the process of moving offices).

Regarding not receiving events at all - we did not manage to reproduce this. Can you try and give us a step by step description of the scenario and how you guys manage to make it happen?

Thank you guys

in the up coming days, not weeks :slight_smile:

There are no solid steps to reproduce the issue where no events show up at all. Please try on different machines, after closing OW and OLP and RL. It has happened to every single one of us, on the latest OW and the latest RL, with the OLP build which you guys have. Hopefully the zip I provided can help with narrowing down what happened to prevent OW from giving us any events.

Right now, between these two issues, we can complete about 1/3 of our matches successfully.

@jeaye, regarding “rosterChange”, we had a mistake and it does work as it should. You can even see it in the RL sample app logs you sent us:

We want to investigate the missing events issue you experience. I’ll ask our support team to contact you as soon as possible

@itayG This issue is saying that we used to get rosterChange and we don’t anymore. The logs you just showed in that picture are from an older conversation between us. I know that because the Steam ids in the logs are not the Steam ids in the logs that I attached to this ticket (i.e. Stealtzkin was not in these recent logs).

So, I’ve reported this issue saying that rosterChange no longer works. Yes, indeed, it used to, which is exactly why we currently rely on it being there. However, if you check the zip I attached to the OP, OverwolfLogs_2019-09-02_14-30-32-no-roster-change.zip, you’ll see that there is no rosterChange at the beginning of the match.

Please investigate the zips I have attached, as well as your own reproduction attempts, rather than referring back to logs which were created when things worked.

The log file was taken from the OverwolfLogs_2019-09-02_14-30-32-no-roster-change.zip file you sent, but I do see it’s an old one.

Can you please try running our RL sample app and send me you latest logs again? I don’t see any new sample app logs for some reason
image

@itayG Just today, we reproduced this issue in the latest OLP build (linked in Slack today). With the app open, we played one game which worked fine, since we got the rosterChange event. Immediately after playing that game, we played another. The second one did not contain a rosterChange event and thus was not valid.

  • The match events submitted at 2019-09-11 22:33:44,204 did contain rosterChange
  • The match events submitted at 2019-09-11 22:35:53,221 did not

This shows the same app, same session, latest OLP build working for one match and then immediately not working. This is the exact reliability issue I’ve been describing and I really hope you can take a look at these logs and find some reason on Overwolf’s side for this inconsistency.

The referenced logs are attached.

OverwolfLogs_2019-09-11_22-36-48.zip (640.1 KB)

Ok, I got it now. I was digging into the logs for some other apps in that zip and found that the rosterChange event did happen for the second match, but it happened before the matchStart event. So the problem here is that we’re expecting rosterChange to happen after matchStart and it does most of the time, but sometimes it happens before.

What are the guarantees on the ordering of events? Which events can be shuffled around? For example, can gameState updates be out of order? Is this a bug or something we need to tolerate? If it’s the latter, this needs to be explicitly documented somewhere, since these events have proven to be anything but deterministic.

Hi @jeaye . I understand that you talked with @LEOkonami.
Once you’ll done please update here so we can move ahead and close the ticket.
thanks.

@eransharv What do you mean? I just posted two updates yesterday. This ticket should not be closed. We need to answer the four questions I ended my last post with to address one of the issues in this ticket. To address the other, we need OW to look at the OverwolfLogs_2019-09-02_14-20-02-no-events.zip zip logs and see why no events showed up at all.

Talking with Or did not fix anything; he had not read this ticket, had not used the app, and the couple games we tried did not reproduce these issues. However, as originally described, these are spurious issues.

  1. Sometimes we get no events from OW at all
  2. Sometimes rosterChange comes in before matchStart

Please answer my four previous questions to help this ticket progress.

@jeaye sorry for the misunderstood. I just wanted to clarify with you if it’s handled on another channel or not. as I know that Or (@LEOkonami) was talking to you on Slack/Email.
In any case, I didn’t intend to close the ticket without getting a confirmation from you.

So, currently, it’s a weekend here. So I will make sure that on Sunday morning the RnD and us will get some answers for all the above. I will update you ASAP.

Thank you for your patience.
Eran

@itayG @LEOkonami FYI.

update: I reopen this ticket. Our customer success manager Asaf will contact you soon.

There has been some confusion here, around the status of this ticket, where the problems exist, and apparently what the problems are. I want to make things super clear to everyone involved. I also want to provide very clear action items regarding what we need from OW at this time.

Firstly, this has been an exploration, since all I knew when reporting these issues was the symptoms and not the cause. What I know now is more than before, but neither of the two problems in this topic have been fully resolved and both require action from OW.

I will now describe each of the two problems we’re seeing.

rosterChange event ordering

Originally, the symptom reported was that the rosterChange event did not show up. However, after further manual inspection of the OW logs, I can see that the event does show up, but not always in the same order. Sometimes it happens before matchStart and sometimes it happens after. This breaks an assumption we were making and it leads to the four questions I asked in a previous message. I will reiterate them here.

  • What are the guarantees on the ordering of events?
  • Which events can be shuffled around?
  • For example, can gameState updates be out of order?
  • Is this a bug or something we need to tolerate?

Answering these questions completely is the first action item for OW. Depending on whether or not this is expected behavior, OW will either need to fix the behavior or update the docs to clearly explain the guarantees provided when it comes to event ordering.

Events not showing up at all

This issue has not been addressed at all. I originally included OverwolfLogs_2019-09-02_14-20-02-no-events.zip to show exactly the state of things just after a match was played, but this has not been addressed. Since OW has not provided any insight into this, I’ve done some analysis on my end and I can see an error reported in the GEP. This error appears to be unique to this zip, so it could very well be the issue.

First, we can examine when the RL game was played.

$ egrep -r -B1 "isInFocus true" Apps/OkLetsPlay/ | tail -2
Apps/OkLetsPlay/index.html.log-2019-09-02 14:11:40,037 (INFO) </Files/profile/overwolf/public/js/app.js> (:249) - INFO [com.okletsplay.front-end.overwolf.event:34] - :onGameInfoUpdated
Apps/OkLetsPlay/index.html.log: #js {:gameInfo #js {:isInFocus true, :isRunning true, :allowsVideoCapture true, :title "Rocket League", :displayName "", :shortTitle "", :id 107981, :classId 10798, :width 1280, :height 720, :logicalWidth 1280, :logicalHeight 720, :renderers #js ["D3D9"], :detectedRenderer "D3D9", :executionPath "C:/Program Files (x86)/Steam/steamapps/common/rocketleague/Binaries/Win32/RocketLeague.exe", :sessionId "c4da0585107b4933aaed1cf15323184a", :commandLine "\"C:/Program Files (x86)/Steam/steamapps/common/rocketleague/Binaries/Win32/RocketLeague.exe\"", :type 0, :typeAsString "Game"}, :resolutionChanged true, :focusChanged false, :runningChanged false, :gameChanged false}

$ egrep -r -B1 "isInFocus false" Apps/OkLetsPlay/ | tail -2
Apps/OkLetsPlay/index.html.log-2019-09-02 14:17:44,475 (INFO) </Files/profile/overwolf/public/js/app.js> (:249) - INFO [com.okletsplay.front-end.overwolf.event:34] - :onGameInfoUpdated
Apps/OkLetsPlay/index.html.log: #js {:gameInfo #js {:isInFocus false, :isRunning false, :allowsVideoCapture true, :title "Rocket League", :displayName "", :shortTitle "", :id 107981, :classId 10798, :width 1280, :height 720, :logicalWidth 1280, :logicalHeight 720, :renderers #js ["D3D9"], :detectedRenderer "D3D9", :executionPath "C:/Program Files (x86)/Steam/steamapps/common/rocketleague/Binaries/Win32/RocketLeague.exe", :sessionId "c4da0585107b4933aaed1cf15323184a", :commandLine "\"C:/Program Files (x86)/Steam/steamapps/common/rocketleague/Binaries/Win32/RocketLeague.exe\"", :type 0, :typeAsString "Game"}, :resolutionChanged false, :focusChanged false, :runningChanged true, :gameChanged false}

So, the game was focused from 2019-09-02 14:11:40,037 to 2019-09-02 14:17:44,475. Just enough time for a 5 minute match. If we grep the GEP logs around that time, specifically for errors, we can see two matches.

$ egrep -r "2019-09-02 14:1{0,8}.*error.*" Apps/Overwolf\ General\ GameEvents\ Provider/
Apps/Overwolf General GameEvents Provider/index.html.log.1:2019-09-02 14:11:43,444 (INFO) </utils/game_plugin.js> (:169) - [PLUGIN INFO] {"category":"plugin_status","key":"state","value":"error","valueLength":5}
Apps/Overwolf General GameEvents Provider/index.html.log.1:2019-09-02 14:11:43,446 (INFO) </utils/game_plugin.js> (:169) - [PLUGIN INFO] {"category":"plugin_status","key":"last_error","value":"failed_initializing_monitor","valueLength":27}

So, curiously, the GEP has an initialization failure in these logs. Following that failure, during the 5 minutes of the RL match, we get absolutely no events. This leads us to an action item for OW:

  1. Please investigate this error and confirm if it would cause us to get no events
  2. Then please inform me if this can be fixed on OW’s end
  3. If this can’t be fixed on OW’s end, please provide some documentation for working around it

Finally, this may be a red herring. I don’t know the internals of OW and my goal in providing these zips on September 2nd was to have someone more knowledgeable inspect them. If this turns out not to be the issue, the next action item for OW is to have someone more familiar with GEP (and also this ticket) work with me to investigate the problem. This leads me to the final point.

The call with Or

There was a call with Or which seemed to’ve caused some confusion regarding whether or not this issue exists, has been fixed, or is on OW’s end rather than our own. Unfortunately, most of this call was unproductive, since Or was neither familiar with this ticket nor our app. Based on what Or said, the community service team uses a separate ticket system and there was no way for him to know about the history of this ticket. I think, going forward, the best usage of everyone’s time is to only schedule calls with people familiar with the issue at hand. To me, it would probably make most sense to be speaking with Eran or Itay.

With that said, Or helpfully provided a debug DLL for the GEP. It will come in handy for any future GEP issues, or perhaps if my analysis of the no-events issue is incomplete or incorrect.

If there are any remaining questions around the status of this ticket, what I’ve found, or what needs to be done, please let me know. I want to be as helpful as possible and all of us at OkLetsPlay are eager to get these issues addressed.

Kind regards,
Jeaye

Hi @jeaye,

I’m so sorry for the confusion and frustration this has caused. As much as we try to provide developers with consistent, up-to-date information, we deeply regret when breakdowns like this occur. And Your feedback is valuable to us.

I refer to each subject on the order of your reply:

rosterChange event ordering

I’m glad that you find out that the rosterChange event does show up.

Regarding the order of appearance, before or after the matchStart - In general, we identify the supported game events in real-time, and immediately fire them on our end. We follow the order that the game fire the events, and we tend not to interfere with this timing.

We don’t have a full mapping regarding the order and timeline of the game events. Our efforts focused on event recognition and identification. We act like kind of a “middle man” between you and the game, which means that we get the event from the game and pass it to you without any logic changes and processing. Having said that, if you are interested in specific info, we will be happy to investigate it.

Events not showing up at all

Regarding the fact that you didn’t get any events in a game session: We inspect the attached logs, and we can see the GEP issue you mentioned from Sep 2th. We can also see in your attached log that on the time of the incident you GEP version was 134.03.

Note that on July 27th, RL released an update that “broke” all the RL events. We announce that on the slack, and a day after that, we released a fixed GEP version (134.04).

So the root cause of the issue was, the fact that on Sep. 2th you still had the old broken GEP version (134.0.3). Probably after a few min or after you finish the game, the GEP automatically updated to the fixed version (134.0.4) and the events back to work again.

The call with Or

Or is responsible for the client support while our team (developers platform team), which includes me, Itay, and Guy, responsible for developers’ support. Because the initial complaint was about the fact that you don’t get any events and we didn’t find any issue, we requested him to check if the matter related to the OW client, to eliminate the causes.

Once again, I apologize for the inconvenience that it caused. I’ve made our team aware of this issue so we can ensure that our different team members always have the right and synced information when a ticket passed between the teams.

Let me know if there’s anything else I can do to help.

Sincerely,
Eran

Thanks for the very helpful response, @eransharv. I think I’m now left with a couple questions regarding the GEP.

  1. Is there any way the OLP app can force that the GEP is always up to date before allowing a match?
  2. Is there a way for OLP to detect the GEP is out of date and require user action?

We cannot leave this sort of thing to chance, so we want to be as proactive about it as possible.

@jeaye there is the new update API. Let me check internally if that can give you the right solution regarding the GEP. I will update in 1-2 days. Thanks.

@jeaye after checking, I found out as I suspected, that the new update API will not update the GEP. The GEP is updated automatically (if an update is available) every time that the OW is restarted.

So, currently, there is no way to check for GEP update on demand. So, we will add an info update that you can listen to Once there is a GEP update; you will be able to notify your users to re-launch the OW client and the game match.

I will update you once it has done (in a few days).