September 29th Service Interruption

First Riot Post
1234511 ... 18
Comment below rating threshold, click here to show it.

Riot tmx

Live Services Manager - Europe

02-10-2013
1 of 8 Riot Posts

Hello Summoners,

This past weekend we had an incident in the EUW service and want to share some details about it and what we learned from it.

We’re sorry – we believe that you deserve the highest quality service and Riot is dedicated to doing better by European players.

What Happened
On September 29th, we experienced unexpected network traffic on our internal servers that was creating abnormally high load on our data caching layer. From your perspective, this may have looked like getting kicked out of the client, sitting in a long login queues, or general slowness in the client navigating between screens. We attempted to lighten this abnormal load by throttling players as they logged onto the service. This inevitably increases the login queue size and in this situation actually exacerbated the issue. At this time, we made the call to restart the service more rapidly return to a stable game experience. Unfortunately, the restart took longer than planned and we did not properly message you on what was happening as it happened.

What we need to do better

  • Communication: It’s crucial that we do a better job communicating with you during an outage. In this case, we didn’t provide regular updates and that was an oversight.
    - We’re committed to regular updates on the service status thread (and @lolstatus on Twitter) until issues are resolved.
  • We also understand that the community craves better communication and community support in general, not just surrounding service issues. We’re committed to this, and we currently have 20 open positions on the community team in Dublin: http://www.riotgames.com/careers?dep...ation%5B%5D=11

Hopefully this gives everyone a bit of clarity on the issue we had this weekend, and how we’re working to improve the quality of the live service.


Comment below rating threshold, click here to show it.

jcv47

Senior Member

02-10-2013

Well, let the communication start with this post, k?
Just give us as much info as possible as early as possible, so we don't start any ranked games, when the servers are laggy ^^
ty


Comment below rating threshold, click here to show it.

Mindspell

Senior Member

02-10-2013

Quote:
Originally Posted by Riot tmx View Post
What we need to do better[LIST][*]Communication: It’s crucial that we do a better job communicating with you during an outage.
Hello,

This communication coming after so many days...

Besides if you know what you need to do to improve the service then why dont you just simply do it?
We seem to already have some community staff on this forum, the problem is that they just wont reply in the important threads, or start threads about correcting the community toxic behavior, as you call it.
They do have time to reply to "how old my account is" and other nonsense like that.
Come on....


Comment below rating threshold, click here to show it.

Sigma117

Senior Member

02-10-2013

I don't fully understand why you're posting this now 3 days after the fact, and during the current server issues.
The information is good, and valid but your timing on this couldn't be worse, providing information about a prior disruption during a current disruption, and saying you need to work on communication during such events is asking for hate.

I think it might've been worth waiting to post this, and doing a larger post about both issues after a solution has been reached.


Comment below rating threshold, click here to show it.

Chager

Associate Producer

02-10-2013
2 of 8 Riot Posts

Hi all,

We also know about the current pain everyone is seeing for the past 2 days with regards to chat, and the fact we've disabled ranked during these chat issues. We've got our top chat and dev guys working on this, and will have a detailed retro like this one for these incidents up tomorrow so you all have insight as to what's going on.

The TLDR for yesterday and today's issues are related to a chat change with the latest release that rearranges your friends list groups. During peak it's causing high load on the chat databases, which is causing issues with chat (friends list not showing up, chat not functioning in champ select, etc). We've disabled ranked during these times of heavy load because chat is an integral part of ranked play. On the back end, we've been adjusting a couple of values to reduce the load on chat while still keeping it enabled, and have a permanent fix we're working on now.

Thanks,
-Chager


Comment below rating threshold, click here to show it.

Sigma117

Senior Member

02-10-2013

Quote:
Originally Posted by Chager View Post
Hi all,

We also know about the current pain everyone is seeing for the past 2 days with regards to chat, and the fact we've disabled ranked during these chat issues. We've got our top chat and dev guys working on this, and will have a detailed retro like this one for these incidents up tomorrow so you all have insight as to what's going on.

The TLDR for yesterday and today's issues are related to a chat change with the latest release that rearranges your friends list groups. During peak it's causing high load on the chat databases, which is causing issues with chat (friends list not showing up, chat not functioning in champ select, etc). We've disabled ranked during these times of heavy load because chat is an integral part of ranked play. On the back end, we've been adjusting a couple of values to reduce the load on chat while still keeping it enabled, and have a permanent fix we're working on now.

Thanks,
-Chager
Repost this as its own thread, and you'll be helping that communication aspect you're wanting to work on. Hiding status reports about a current issue in a thread about a prior one will make it more likely to be missed.


Comment below rating threshold, click here to show it.

gomuninguen

Senior Member

02-10-2013

>1 month left to get gold >Riot disables rankeds everyday

Thanks a lot...


Comment below rating threshold, click here to show it.

Lynléire

Senior Member

02-10-2013

Quote:
Originally Posted by Sigma117 View Post
Repost this as its own thread, and you'll be helping that communication aspect you're wanting to work on. Hiding status reports about a current issue in a thread about a prior one will make it more likely to be missed.
Gotta agree with this!


Comment below rating threshold, click here to show it.

abuzeus

This user has referred a friend to League of Legends, click for more information

Member

02-10-2013

Quote:
Originally Posted by Riot tmx View Post
(and @lolstatus on Twitter)
That twitter account is useless. Currently it is reporting that the NA fora will be unavailable. On Sunday it was not used. It takes hours until someone actually tweets something on it (like on Monday).

The last time there was a big incident (end of May/early June) you identified communication as a top priority, you promised to have canned scripts and posts ready. I wanted to believe, but it has just not happened. Every single of the last days, there has been a red notice on reddit for hours until a service status post went up. I really don't understand why it takes so long. Especially after several days with problems in succession, you need to monitor third party sites, not just your statistics. At most 30 minutes after people start to complain, you need to post something, even if you can not verify that there is a problem from your side. Because, generally there has been a problem, and I really don't understand what's so hard about having a text file sitting somewhere saying

"Summoners, we have been getting reports of disconnects/chat issues/login queue/real life Teemo sightings and are investigating. We will update this thread and report back to you in at most an hour."

I am really disappointed, and I do consider the lackluster communication a broken promise. Riot makes so much cool stuff, and I really love this game, but the repeated concentration of problems on my home server and the sub par service here makes it sooo hard (not to mention a bunch of other things that make me angry, but are out of scope here). I mean, people probably work really hard to keep the servers afloat, but somewhere some management guy makes a lot of questionable decisions.

Also you need to extend the timeframe for ranked season by at least a week, and it needs to be EUW exclusively. I feel that this is very much warranted given that ranked was unavailable so much during peak times.


Comment below rating threshold, click here to show it.

Chager

Associate Producer

02-10-2013
3 of 8 Riot Posts

@Sigma117 & Lynléire

Y'all raise a very valid point, I'll create a separate thread shortly with info on today's incident.

A unique challenge we face with posting in EUW is that we have 7 different languages that we support, and don't want any of them to feel less valued than the others. The translating adds a complication when you're trying to share up to the minute information. That being said, I totally agree that communication sooner is more valuable than perfectly translated messaging after the fact.

Feedback taken.


1234511 ... 18