To get to the castle:

Enter a location when the message indicates you are being watched from that location (giggles, etc). That will trigger a boss battle for the location. Defeat the bosses of either Forest or Grove, then Swamp and Deep Forest. Each boss gives you a map fragment. You should now ...

Hi, really liked the club, still cant find the scarecrow...
Can someone tell me how to find the Castle? I still dont understand how to get there.