[News, Bugs, War, Failure Protocol] Dev Response ( Quoted here for visibility )

[Gryphonknight]

Subject
[News, Bugs, War, Failure Protocol] Dev Response ( Quoted here for visibility )

@Sara

Regarding the Issues with Alliance Wars and Heroes with Minions

Here is a short update and a big apology for the issues that many of you faced during the Alliance Wars since we launched version 1.12.5 on Tuesday the 29th of May.

What Happened?

Despite our internal testing procedures, we didn’t notice that the old minions had stopped inflicting damage to the enemies.

Normally we can run two versions of the game simultaneously. This applies to Alliance Wars too, as long as the rules of the battle haven’t changed. Unfortunately the bug did change the rules.
When players started to encounter problems with Alliance Wars, we first fixed the problem on Wednesday by forcing everybody to update to version 1.12.5.

However, we naturally wanted to fix the original problem too. We implemented version 1.12.6 which fixes the minions. Because this changes the battle logic, we would have needed to force another version change.
Due to the way Apple and Google cache updates, these forced version changes always cause a lot of problems to our users. To avoid a forced update, we made the decision to support both 1.12.5 and 1.12.6 at the same time on the same server. This was a mistake. While normal battles worked correctly, Alliance Wars did not.

Unfortunately the weekend war starts on a Saturday night here in Helsinki. As you may know, we are a relatively small company of about 30 people. So far we have been relying on automated alerts to notify our on-call person about potential problems during nights and weekends. While our staff does browse the forums and play the game on their spare time, they are unfortunately not able to provide the same level of support they do during the office hours.

In this case, there was some increase in the amount of warnings on the server, but nothing drastic enough to trigger an automatic alert.

We learned about the problems on Sunday morning, and implemented a fix while there was still 12 hours of the war remaining. Unfortunately the fix worked only partially. If the battle failed in a conflict we’d detect it and rerun the battle with the other logic. As soon as we deployed the fix, the conflicts disappeared. What we did not realise is that there were many battles that didn’t produce a conflict at all - they had only valid moves, but due to the version difference, the end result of the battle was different on the client and the server.

Despite our best efforts we were unfortunately unable to fully resolve the issue before the war was over.

What Now?

• We’ve sent an in-game message to all the players eligible for Alliance Wars. The message includes a small donation as an apology

• We are going to make sure this bug is fixed before the next Alliance War

• We’ve held a a retrospective about the incident, and will implement a number of technology and process improvements to prevent a similar issue from happening again. As a concrete example, we will change the Alliance War battle validation to detect the situation where the state between the server and the client is different, even though the battle itself completed without a conflict.

Again, we sincerely apologise about the issues. We will do better in the future! Thank you for reading, and good luck in the Battles!

@Rook, @Coppersky, please lock this topic so people reply on the original topic. Thank you.

[Coppersky]

To add your thoughts on the above post, click on link in above post to get to original thread.