For some reason whenever I go to my chosen location to fight - in this case Phantom Warriors in Seemlia - I have one battle and then I go to hero's screen to drink a potion or something - it automatically takes one more AP - therefore I lose 1 for the battle that occured and 1 AP just whenever I go to Hero, without fighting further.
I tested it 3 times so I am pretty sure its not just an illusion. Noone has attacked me and I am not losing further hitpoints, just AP.