Basically on defense, the enemy hero slash attack and minion attacks do take the element into consideration when there is a EDD or EDU
Special attacks are neutral unless specifically stated to do extra damage like GM vs Greens for example
Edit : Some extra clarification about special skills. While the skills themselves are neutral, if you have elemental def down of a specific color, then the damage of special skills from heroes of that color are boosted
