Rocket League event regression

We want to investigate the missing events issue you experience. I’ll ask our support team to contact you as soon as possible

@itayG This issue is saying that we used to get rosterChange and we don’t anymore. The logs you just showed in that picture are from an older conversation between us. I know that because the Steam ids in the logs are not the Steam ids in the logs that I attached to this ticket (i.e. Stealtzkin was not in these recent logs).

So, I’ve reported this issue saying that rosterChange no longer works. Yes, indeed, it used to, which is exactly why we currently rely on it being there. However, if you check the zip I attached to the OP, OverwolfLogs_2019-09-02_14-30-32-no-roster-change.zip, you’ll see that there is no rosterChange at the beginning of the match.

Please investigate the zips I have attached, as well as your own reproduction attempts, rather than referring back to logs which were created when things worked.

The log file was taken from the OverwolfLogs_2019-09-02_14-30-32-no-roster-change.zip file you sent, but I do see it’s an old one.

Can you please try running our RL sample app and send me you latest logs again? I don’t see any new sample app logs for some reason
image

@itayG Just today, we reproduced this issue in the latest OLP build (linked in Slack today). With the app open, we played one game which worked fine, since we got the rosterChange event. Immediately after playing that game, we played another. The second one did not contain a rosterChange event and thus was not valid.

  • The match events submitted at 2019-09-11 22:33:44,204 did contain rosterChange
  • The match events submitted at 2019-09-11 22:35:53,221 did not

This shows the same app, same session, latest OLP build working for one match and then immediately not working. This is the exact reliability issue I’ve been describing and I really hope you can take a look at these logs and find some reason on Overwolf’s side for this inconsistency.

The referenced logs are attached.

OverwolfLogs_2019-09-11_22-36-48.zip (640.1 KB)

Ok, I got it now. I was digging into the logs for some other apps in that zip and found that the rosterChange event did happen for the second match, but it happened before the matchStart event. So the problem here is that we’re expecting rosterChange to happen after matchStart and it does most of the time, but sometimes it happens before.

What are the guarantees on the ordering of events? Which events can be shuffled around? For example, can gameState updates be out of order? Is this a bug or something we need to tolerate? If it’s the latter, this needs to be explicitly documented somewhere, since these events have proven to be anything but deterministic.

Hi @jeaye . I understand that you talked with @LEOkonami.
Once you’ll done please update here so we can move ahead and close the ticket.
thanks.

@eransharv What do you mean? I just posted two updates yesterday. This ticket should not be closed. We need to answer the four questions I ended my last post with to address one of the issues in this ticket. To address the other, we need OW to look at the OverwolfLogs_2019-09-02_14-20-02-no-events.zip zip logs and see why no events showed up at all.

Talking with Or did not fix anything; he had not read this ticket, had not used the app, and the couple games we tried did not reproduce these issues. However, as originally described, these are spurious issues.

  1. Sometimes we get no events from OW at all
  2. Sometimes rosterChange comes in before matchStart

Please answer my four previous questions to help this ticket progress.

@jeaye sorry for the misunderstood. I just wanted to clarify with you if it’s handled on another channel or not. as I know that Or (@LEOkonami) was talking to you on Slack/Email.
In any case, I didn’t intend to close the ticket without getting a confirmation from you.

So, currently, it’s a weekend here. So I will make sure that on Sunday morning the RnD and us will get some answers for all the above. I will update you ASAP.

Thank you for your patience.
Eran

@itayG @LEOkonami FYI.

update: I reopen this ticket. Our customer success manager Asaf will contact you soon.

There has been some confusion here, around the status of this ticket, where the problems exist, and apparently what the problems are. I want to make things super clear to everyone involved. I also want to provide very clear action items regarding what we need from OW at this time.

Firstly, this has been an exploration, since all I knew when reporting these issues was the symptoms and not the cause. What I know now is more than before, but neither of the two problems in this topic have been fully resolved and both require action from OW.

I will now describe each of the two problems we’re seeing.

rosterChange event ordering

Originally, the symptom reported was that the rosterChange event did not show up. However, after further manual inspection of the OW logs, I can see that the event does show up, but not always in the same order. Sometimes it happens before matchStart and sometimes it happens after. This breaks an assumption we were making and it leads to the four questions I asked in a previous message. I will reiterate them here.

  • What are the guarantees on the ordering of events?
  • Which events can be shuffled around?
  • For example, can gameState updates be out of order?
  • Is this a bug or something we need to tolerate?

Answering these questions completely is the first action item for OW. Depending on whether or not this is expected behavior, OW will either need to fix the behavior or update the docs to clearly explain the guarantees provided when it comes to event ordering.

Events not showing up at all

This issue has not been addressed at all. I originally included OverwolfLogs_2019-09-02_14-20-02-no-events.zip to show exactly the state of things just after a match was played, but this has not been addressed. Since OW has not provided any insight into this, I’ve done some analysis on my end and I can see an error reported in the GEP. This error appears to be unique to this zip, so it could very well be the issue.

First, we can examine when the RL game was played.

$ egrep -r -B1 "isInFocus true" Apps/OkLetsPlay/ | tail -2
Apps/OkLetsPlay/index.html.log-2019-09-02 14:11:40,037 (INFO) </Files/profile/overwolf/public/js/app.js> (:249) - INFO [com.okletsplay.front-end.overwolf.event:34] - :onGameInfoUpdated
Apps/OkLetsPlay/index.html.log: #js {:gameInfo #js {:isInFocus true, :isRunning true, :allowsVideoCapture true, :title "Rocket League", :displayName "", :shortTitle "", :id 107981, :classId 10798, :width 1280, :height 720, :logicalWidth 1280, :logicalHeight 720, :renderers #js ["D3D9"], :detectedRenderer "D3D9", :executionPath "C:/Program Files (x86)/Steam/steamapps/common/rocketleague/Binaries/Win32/RocketLeague.exe", :sessionId "c4da0585107b4933aaed1cf15323184a", :commandLine "\"C:/Program Files (x86)/Steam/steamapps/common/rocketleague/Binaries/Win32/RocketLeague.exe\"", :type 0, :typeAsString "Game"}, :resolutionChanged true, :focusChanged false, :runningChanged false, :gameChanged false}

$ egrep -r -B1 "isInFocus false" Apps/OkLetsPlay/ | tail -2
Apps/OkLetsPlay/index.html.log-2019-09-02 14:17:44,475 (INFO) </Files/profile/overwolf/public/js/app.js> (:249) - INFO [com.okletsplay.front-end.overwolf.event:34] - :onGameInfoUpdated
Apps/OkLetsPlay/index.html.log: #js {:gameInfo #js {:isInFocus false, :isRunning false, :allowsVideoCapture true, :title "Rocket League", :displayName "", :shortTitle "", :id 107981, :classId 10798, :width 1280, :height 720, :logicalWidth 1280, :logicalHeight 720, :renderers #js ["D3D9"], :detectedRenderer "D3D9", :executionPath "C:/Program Files (x86)/Steam/steamapps/common/rocketleague/Binaries/Win32/RocketLeague.exe", :sessionId "c4da0585107b4933aaed1cf15323184a", :commandLine "\"C:/Program Files (x86)/Steam/steamapps/common/rocketleague/Binaries/Win32/RocketLeague.exe\"", :type 0, :typeAsString "Game"}, :resolutionChanged false, :focusChanged false, :runningChanged true, :gameChanged false}

So, the game was focused from 2019-09-02 14:11:40,037 to 2019-09-02 14:17:44,475. Just enough time for a 5 minute match. If we grep the GEP logs around that time, specifically for errors, we can see two matches.

$ egrep -r "2019-09-02 14:1{0,8}.*error.*" Apps/Overwolf\ General\ GameEvents\ Provider/
Apps/Overwolf General GameEvents Provider/index.html.log.1:2019-09-02 14:11:43,444 (INFO) </utils/game_plugin.js> (:169) - [PLUGIN INFO] {"category":"plugin_status","key":"state","value":"error","valueLength":5}
Apps/Overwolf General GameEvents Provider/index.html.log.1:2019-09-02 14:11:43,446 (INFO) </utils/game_plugin.js> (:169) - [PLUGIN INFO] {"category":"plugin_status","key":"last_error","value":"failed_initializing_monitor","valueLength":27}

So, curiously, the GEP has an initialization failure in these logs. Following that failure, during the 5 minutes of the RL match, we get absolutely no events. This leads us to an action item for OW:

  1. Please investigate this error and confirm if it would cause us to get no events
  2. Then please inform me if this can be fixed on OW’s end
  3. If this can’t be fixed on OW’s end, please provide some documentation for working around it

Finally, this may be a red herring. I don’t know the internals of OW and my goal in providing these zips on September 2nd was to have someone more knowledgeable inspect them. If this turns out not to be the issue, the next action item for OW is to have someone more familiar with GEP (and also this ticket) work with me to investigate the problem. This leads me to the final point.

The call with Or

There was a call with Or which seemed to’ve caused some confusion regarding whether or not this issue exists, has been fixed, or is on OW’s end rather than our own. Unfortunately, most of this call was unproductive, since Or was neither familiar with this ticket nor our app. Based on what Or said, the community service team uses a separate ticket system and there was no way for him to know about the history of this ticket. I think, going forward, the best usage of everyone’s time is to only schedule calls with people familiar with the issue at hand. To me, it would probably make most sense to be speaking with Eran or Itay.

With that said, Or helpfully provided a debug DLL for the GEP. It will come in handy for any future GEP issues, or perhaps if my analysis of the no-events issue is incomplete or incorrect.

If there are any remaining questions around the status of this ticket, what I’ve found, or what needs to be done, please let me know. I want to be as helpful as possible and all of us at OkLetsPlay are eager to get these issues addressed.

Kind regards,
Jeaye

Hi @jeaye,

I’m so sorry for the confusion and frustration this has caused. As much as we try to provide developers with consistent, up-to-date information, we deeply regret when breakdowns like this occur. And Your feedback is valuable to us.

I refer to each subject on the order of your reply:

rosterChange event ordering

I’m glad that you find out that the rosterChange event does show up.

Regarding the order of appearance, before or after the matchStart - In general, we identify the supported game events in real-time, and immediately fire them on our end. We follow the order that the game fire the events, and we tend not to interfere with this timing.

We don’t have a full mapping regarding the order and timeline of the game events. Our efforts focused on event recognition and identification. We act like kind of a “middle man” between you and the game, which means that we get the event from the game and pass it to you without any logic changes and processing. Having said that, if you are interested in specific info, we will be happy to investigate it.

Events not showing up at all

Regarding the fact that you didn’t get any events in a game session: We inspect the attached logs, and we can see the GEP issue you mentioned from Sep 2th. We can also see in your attached log that on the time of the incident you GEP version was 134.03.

Note that on July 27th, RL released an update that “broke” all the RL events. We announce that on the slack, and a day after that, we released a fixed GEP version (134.04).

So the root cause of the issue was, the fact that on Sep. 2th you still had the old broken GEP version (134.0.3). Probably after a few min or after you finish the game, the GEP automatically updated to the fixed version (134.0.4) and the events back to work again.

The call with Or

Or is responsible for the client support while our team (developers platform team), which includes me, Itay, and Guy, responsible for developers’ support. Because the initial complaint was about the fact that you don’t get any events and we didn’t find any issue, we requested him to check if the matter related to the OW client, to eliminate the causes.

Once again, I apologize for the inconvenience that it caused. I’ve made our team aware of this issue so we can ensure that our different team members always have the right and synced information when a ticket passed between the teams.

Let me know if there’s anything else I can do to help.

Sincerely,
Eran

Thanks for the very helpful response, @eransharv. I think I’m now left with a couple questions regarding the GEP.

  1. Is there any way the OLP app can force that the GEP is always up to date before allowing a match?
  2. Is there a way for OLP to detect the GEP is out of date and require user action?

We cannot leave this sort of thing to chance, so we want to be as proactive about it as possible.

@jeaye there is the new update API. Let me check internally if that can give you the right solution regarding the GEP. I will update in 1-2 days. Thanks.

@jeaye after checking, I found out as I suspected, that the new update API will not update the GEP. The GEP is updated automatically (if an update is available) every time that the OW is restarted.

So, currently, there is no way to check for GEP update on demand. So, we will add an info update that you can listen to Once there is a GEP update; you will be able to notify your users to re-launch the OW client and the game match.

I will update you once it has done (in a few days).

@jeaye @Bilekill FYI - I created a feature request for the R&D.
I asked them to implement a new API function: checkForGEPUpdate().

This way, you will be able to check on demand if there is a GEP update. If so - you can notify the user to restart Overwolf and restart the game, in order to make sure that all the events are working properly.

I remind you that anyway every OW restart and every couple of hours, the GEP is updated automatically.

Internal link to the R&D FR: xxx.com/agiles/93-44/97-1682?issue=PLAN-310

@eransharv That’s great news! To be clear, does the user need to restart the OLP app, or all of Overwolf? If the user needs to restart all of OW, it may be a good idea to have an API fn to do that. This will allow us (and every other dev) an “Update now” button which will hit the new update API to update OLP and then check for GEP updates before automatically restarting OW (assuming updates were found).

Do you agree this is the best UX?

@jeaye hi.

We rethink about it, and we decided to implement it on our infrastructure. Means,
On the next few versions, OW will automatically update the GEP every time a game is launched or when the player go back to the lobby. (not in a middle of a match of course).

So i will go ahead and mark this ticket as closed. Please let me know for any other question.
Thanks.

Can we please not close this ticket? The ticket is a symbol of the issue OLP has and that issue is not resolved. Once OW comes out with a version which auto-updates the GEP before every gaming session, we could possibly mark this resolved.

However, there is still the possibility where I started my game 4 hours ago, prior to a GEP release, and then my opponent just starts his to play me. His GEP updates, but my game is already open, so mine doesn’t. If your changes won’t address this then we absolutely need the ability to check the GEP version of both players so we can require one of them to restart the game, if needed.

Sure i can leave it open till than. I don’t have the ETA yet for this feature.

Once the feature will be implemented, your GEP will be updated every time that you get to the lobby.
So even if the game is open for hours, from time to time when the player go back to the lobby, the GEP will get his chance to be updated.

@itayG FYI

Hey guys. It’s been another month and OkLetsPlay has launched with its real money support, even without this ticket being addressed. However, that puts us at serious risk, since we can’t rely on players actually getting their events. They may have an outdated GEP. We also have no API for confirming whether or not two players have the same GEP version.

To work around this for our hard launch, we have made it so that we only require results from one player, since the other may have an outdated GEP. This doesn’t handle all cases, since both players may have an outdated GEP, in which case we’ll get no events. However, the biggest issue with it is that it’s entirely insecure, since we can’t trust the results from just one player; we need to have the results from both players so we can confirm they are consistent. This is made much worse with team-based matches, since now we’ll be looking at 6 people’s results (3v3) and we need to be sure we’re going to get all of them.

This is a serious issue and it will spell the demise of any apps relying on Overwolf for anything consequential, like our real money matches. Please address this and let me know how I can help; literally, if you want us to help you work on this code, that’s something we can discuss.

We need this feature.