Against Finley, hold your Turtle banner for after he fires his skill, so you overwrite the negative defense ailment.
In general, it’s usually better to increase your defense to protect yourself from attacks. The regular slash attacks from enemies is multiplied by 100, so they overall deal more damage when they attack you.
Further reading: Damage Calculation