Damn these guys are sneaky. I had built a wall and keep with one-way doors to keep the raiders out. Anyhow, I'm working away and one of them attacks me from inside the keep. You have an advantage when fighting on your own ground so I was able to kill him off. Luckily he attacked before he could call in his mates.
Ends up he had been digging holes around my claimed land and had found a natural cave that eventually led into my keep.
So the next thing I learned was to make sure ALL entrances (above and below ground) are sealed with a door positioned in your side of the border.
Another really cool part of the mod (or it may be another mod) is that as you perform tasks you go up in skill and get bonuses as your skill increases. For example, if you are mining ore and have a reasonable mining skill level, some of the raw stone around where you are mining may change into the ore you are mining. Really nice touch.