Post Mortem on the Server Downtime on the 12th of August

Thanks for taking the time to explain this to your “customer” base. It really is much appreciated.
More of this type of open communication from SG would in my opinion, be immensely valuable.

10 Likes

Yes - these things happen. It’s also important to keep the communication channel open with the customers, and promptly taking care of such issues as they arise.

Thank you for doing all of that and more, @mhalttu! Much appreciated!! :slightly_smiling_face:

5 Likes

thanks for the explanation

hopefully they cut the blue wire next time

@mhalttu does this mean we’re
not gettin anymore free stuff? askin for a friend lol

4 Likes

probly won’t ever get an answer to this but really curious why war wasn’t cancelled when it has been cancelled in the past for smaller setbacks

16 Likes

Thanks for the clarification, @mhalttu. I truly appreciate SGG’s communication with us. As some colleagues have already mentioned, these things happen. Nothing is foolproof, but it is good to have an open communication channel, to avoid turmoil and exaggerated speculation. Thanks again for the attention and work of the team in solving the problem.

4 Likes

Feedback from the company is always appreciated. Thank you for taking the time to respond.

May we see more feedback in the future?

8 Likes

Appreciate the communication. Would appreciate an answer to why wars wasn’t cancelled @mhalttu.

3 Likes

If I understand correctly then, updating the Mongo DB version would fix this somewhat? I’m reading through the Jira ticket and it sounds like the problem has been fixed for the most part (says that it will still affect queries that need to access any secondaries though).

Forgive me if I sound ignorant with Mongo – most of my database development has been in SQL Server in my career so far, and I’ve just never seen index problems like this when involving replication in SQL (it has its own share of replication gotchas too though).

Regardless, thanks for sharing. You definitely didn’t have to do that with us, but it’s very appreciated.

7 Likes

@mhalttu. Thanks for letting us know what happened. There are a lot of us that really care about the game perhaps some of us a little too much .

All of you asking about the war, I don’t think canceling it or not is a decision that falls under IT.

3 Likes

They’ve done it before, what’s changed? I’m curious to know why.

2 Likes

@Darth_Vitiate: Do you ask you IT guys at work to approve your OT or extend your project deadline? No, because they don’t make those decisions.

There is likely someone in charge of customer experience or development that can cancel wars or give gifts to players. BTW, thanks for the event coins EandP!

2 Likes

I pulled Finley with ten of those.

4 Likes

Thank you @mhalttu for this feedback, the communication is very much appreciated and thanks to the rest of your team too for the hard work in bringiing those servers back to their expected working states…hopefully the learned points would forestall similar issue in future.

Probably not. But this is the only robust direct communication that we’ve seen on the topic. I’d expect that if @mhalttu isn’t the one to answer it, he does have a way to say to the right person ‘Hey, there are a lot of questions on this. You should offer an explanation from your side of things.’

It’s not like other SG staff can’t add their 2 cents.

2 Likes

He’s the CTO and normally is involved in making calls like war cancellation :slightly_smiling_face:

6 Likes

He was the one who polled for and cancelled the 1/2* raid tournaments. Rumour has it when he returned home from SG Towers that day, Layla had cut up all his ties and Hou had done a :poop: on his lawn.

9 Likes

While I have no idea what the heck you just said, I’m not new to online games, and they do have outages. Many games have planned 2-3 hour long outages every week (not including the occasional unplanned ones). Got to give you guys kudos for being able to keep the game up and running while doing maintenance / updates.

The only thing I wanted to know on that day was what was going to happen regarding titans / wars etc. Was quite content with the compensation after the fact. And with this very technical explanation. Appreciate the transparency. :wink:

As I said… I’m a longtime gamer. Server stuff happens, it’s normal. We would just like to be kept somewhat in the loop though.

An example for comparison… occasionally my water or power or internet or whatever gets cut off. I call in to report it and ask about it… if they tell me, “sorry, there’s an issue, we’re working on it”… okay… do you know when it will be fixed? “Not exactly, but we estimate 2-4 hours”… okay. I’ll call back again in 4 hours if it’s still down.

If instead I call the company and I get, “our offices are currently closed” or “all lines are busy”… that’s annoying. :angry: A simple prerecorded message acknowledging the issue and indicating that it’s being addressed is very much appreciated.

But anyway… thank you guys for the compensation, and for explaining the issue. It’s unfortunate that it happened on a war day, leaving many of us “in the dark”, but… (**)it happens. :man_shrugging: :wink:

7 Likes

@mhalttu Wow, way more info than expected or needed. Thanks to you and the tech team for the diligence.

1 Like

lol

so what you’re tryin to say is “Rigs is right”

goin to go ahead and mention mhalltu was also the guy to flip a switch and change war penalties from 10 to 20 in the blink of an eye…

4 Likes

thanks @mhalttu, I really apreciated your detailed high level report and really really do sympathize with you and your team for what happened.

working in replicated database environment myself, I do know how bad things may turn when it breaks and the cluster goes down, even if I’ve never deployed solutions over mongodb (rather on sql server, being a certified microsoft database expert and solution developer), it always gets my attention seeing how some unlucky events may happen to any dbms environment out there.

you did a great job in bringing the system back in so little time and recovering all the lost transactions while keeping us updated.

thanks for the 20 coins gift. :slight_smile:

3 Likes

Cookie Settings