Post Mortem on the Server Downtime on the 12th of August

I stand corrected.

Commence demanding answers. lol

3 Likes

Thanks for the detailed explanation. Great work always

1 Like

No one’s been “demanding” any answers here. And least from you.

Good day to you.

3 Likes

Unfortunately, that phrase is banned under forum rules :pensive:

If I ever admit you’re right I’d immediately have to flag myself.

10 Likes

did you have to approve your own post for usin the B word?

1 Like

Funnily enough, I didn’t this time :thinking:

But I’m tangentially misdirecting the thread

  • horray for clarity
  • boo to server error
  • war decision rationale would be interesting
5 Likes

Well this is a first. I’m actually going to give SG more credit than you did.

  • horray for clarity

+1 to this. I like it when game companies own up to their issues.

  • boo to server error

Actually going to give SG a pass on that.

WTF TGW? You defending SG now? Traitor!!!

Yep. On the issue of the outage itself? I am defending them.

As I’ve said many times before. I’ve played many video games over many years. They all have outages. Overall? SG probably has fewer outages than most of the other online games I’ve played.

Yeah, I’m critical, but I’m also honest. Have to give credit where credit is due. SG servers are more reliable in terms of total weekly “uptime” vs. many other games. Occasional downtimes are to be expected.

  • war decision rationale

This was really the main part that bothered me.

I mean, I guess I get it… on the one hand, if they cancelled the war, that would screw up PoV progress, so then they’d probably have to go in later and do something about that. Plus lots of people were looking forward to / preparing for the war.

The outage probably skewed the scores for some alliances. But even so… a lost war gives more loot than a cancelled war. 1 point in war chest is more than 0 points in war chest. Credit for 6 war flags used towards PoV is better than 0. Trying to go in after the fact to fix every possible scenario individually would have been a major headache, and they already had enough of a headache dealing with what they were already dealing with.

Worst case scenario in allowing the war to continue? One team gets screwed into a loss that could have been a win. Well… that could have happened anyway without a worldwide outage, couldn’t it? In retrospect, I realize now I shouldn’t have been so upset about it. So we lost. So we drop down a tier. So the next war will be easier. And it was. I was mostly just annoyed by the lack of communication at the time.

“Let the war continue!”, in hindsight, was probably actually the best / most effective overall decision. Just with a nice little asterisk behind the war results, like when a sports champion has been caught with a locker full of performance enhancing drugs…

“Oh yeah sure, you won, but you didn’t exactly win under ‘fair’ conditions. So enjoy your record and your win. Just remember that it doesn’t really count.” :laughing:

9 Likes

Completely agree. I understood the explanation because my husband is a Unix admin and working talks to me about things. I do wish for more transparency like this.

1 Like

That explains all the flags.

4 Likes

First off, thank you @mhalttu for the detailed explanation. I did understand it, although I’m annoyed that MongoDB requires a JIRA login just to see a ticket (that may be a JIRA thing, which is another reason to dislike it).

Anyway…

Nailed it. That’s what’s different now compared to earlier times. POV. Plain and simple.

A post was split to a new topic: Account recovery :es:

There have been questions about why the war wasn’t cancelled.

We did discuss the topic. We try to avoid cancelling a war if possible. Usually when we do, it’s either because a large number of players have been affected over a longer period of time or there is a bug that would cause problems throughout the war.

In this case, the war had just began at the time of the backup, 10:33 UTC. The only people who were directly affected were the ones who had started the attack right before the backup (because the result of their attack would be lost). Of course, everybody else was affected by the downtime, but at least they were affected equally.

In a situation like this, you have to make quick decisions. You discuss and try to make the best decision with the information and the time you have. For the reasons outlined above, we made the decision not to cancel the war.

If you think there is a strong reason why it was a mistake, I would like to hear it so we can take it into account when making similar decisions in the future.

25 Likes

Can you tell us, how many players have lost their war flag from this incidents?

The strong reason is a loss by 15 points with a 50 point hit counted as a zero.

The point is that if results are effected then counting the war apples to apples with other wars is wrong. A better solution is to also rollback the war to the same point the entire rest of the games was rolled back to and stop the war clock throughout the outage. That way all wars actually last 24 hours and full flags are available for each team. It is inconceivable that war is the only part of the game not subject to a rollback.

Looking at the data recovery efforts, kudos to finding all the lost 5 star heroes and 4 star mats. Why stop there? Please find all the lost war flags!

6 Likes

Not 100%… I lost 2 war flags counting toward PoV due to the roll back. Submitted a ticket and was told “sorry… there’s still plenty of time.” (And I would NEVER leave a war flag on the field anyway…) Guess it’s better than losing 6 but goes to the point that parts of the game were rolled back and others were not…

3 Likes

Am I the only one whose alliance actually won that war and didn’t waste any flags?

That’s messed up. I understand it’s a small thing, but the rollback should have rolled back FULLY (war attacks undone, flags given back) or they should fix the small things that fell through the cracks, like POV counters.

As they restored only one table, and that table apparently has POV data, they get to make tweaks for anyone that actually files a ticket on it.

But that’s my opinion, and carries no weight.

3 Likes

panda covered most of it

but I’d like to point out that not every single player in every time zone was effected the same way

i can’t say it with 100% certainty as I’m not in the opponents team we had for war so don’t know their strat

but it seemed they were done using flags before the outtage happened and their war strat was uninterrupted since the outtage was fixed just in time for the respawn of their farm targets, where as us being mainly North America, our war strat was decimated and our team had to do a whole new plan on the fly

basically what our opponents do was backwards from us in a sense. what they do in the middle of war, we do at the beginning. why? best availability but due to the outtage being at the worst time for us and an irrelevant time for them, i "believe " i can say that no, not everyone was affected equally by that downtime

rolling war clocks back to the time they were when outtage occured(if even doable) would have been the most fair solution i believe

4 Likes

Perhaps the best TL;DR in forum history.

More excellent, succinct analysis. Am I on the right boards?

This kind of communication from the devs is very welcomed, @mhalttu. Thanks for dropping in not once, but twice, to address concerns. More of same, please.

9 Likes

Hope you didn’ t fire the guy, it’s hard to get a job in this Covid 19 situation