Testing indicated that the counter activates first (full damage countered), and then afterwards the damage taken is reduced.
This is because the damage reduction is not calculated like a regular defense buff. In the damage formula:
100 x (1/3 if tile damage, otherwise 1) x (θ x Att / Def)^1.35
The total attack value is divided by the total defense value. The defense buffs are applied in the bottom part of the division.
The formula above is what is used to calculate the countered damage. That’s why defense buffs and damage share buffs affect countered damage negatively, since the damage taken gets reduced in that formula.
Damage reduction, however, is applied after the entire formula is done.
In other words:
Att / Def (defense buffs calculated here)
( 100 x (1/3 if tile damage, otherwise 1) x (θ x Att / Def)^1.35 ) x damage reduction modifier after entire formula is done