I don't wear armour. I might on a PVP server, but it's rarely worth it on non-PVP SMP. Occasionally when I'm in my farm I'll slaughter the cows. And occasionally once I have enough leather I'll make a set of leather armour. But it's more for show than anything. It won't save you from fire/lava at all.
It'll stop monsters hurting so hard, but you'd have to be in a fairly dangerous place to take that many hits at once. Like Bazz said, when exploring natural cave systems (with this new mob spawn rate which makes larger groups) is probably a good idea to have some armour on. Went exploring with a few folks the other day and got blown up by creepers 8 times in 3 minutes.
For general mining or building, I wouldn't worry too much.
PS. And no - do not build a diamond armour set. You'll regret it very much. Even if you happen to not fall into lava and lose it, that's a lot of pickaxes you won't have had - and it takes a long time to find that amount of diamond. I've probably found 50 diamond "in total" so far. Sure, I could've made a suit out of it, but it's been put to far better use in the 15 or so Diamond Pickaxes I've made.