Ok, I'm going to go on a completely different track to answer this question.
The first game to scare me? Eternal Darkness: Sanity's Requiem on the Nintendo Gamecube.
For those of you unfamiliar with the game, you had your typical health and magic meters. Eternal Darkness added a third gauge: a Sanity meter. Essentially, being exposed to zombies and such slowly lowered your character's sanity. To get it back, you'd have to kill creatures.
As your sanity dropped, small strange things began to happen in the game. You would see a book fly off a shelf onto another. A statue's head would follow you as you walked passed it, but would not do it again a second time. You'd hear knocking or ringing coming from a room, and there would be nothing in there.
As the sanity meter further depleted, the game began playing some nasty tricks on you. You would walk into a room only to find yourself three rooms back. The noises would become more prominent. Heck, even after finishing a chapter midway through the game, it gave me a "The End" screen.
As the gauge went to its lowest levels, the screen would tilt at an odd angle, the music would get really creepy, and you'd hear constant evil laughter. This is when the game would really start to screw with you and broke the 4th wall.
I remember walking into a room with a half dozen zombies in it. My sanity meter was rather low, but I knew I'd have no problem against these guys. As they shuffle towards me, a message suddenly pops up on screen: "The controller has been disconnected from the system. Please reconnect the controller cord." For those of you who had a Gamecube, this was an all too common occurrence, but it was happening at the worst possible time. I bolt from the couch, furiously plugging and unplugging the controller, all while my character is being decimated. The message won't go away, I have no control of my character, and eventually she is killed by the zombies. The game over screen comes on, and I am filled with dread as I realize my last save was over an hour ago.
The Game Over screen disappears. I'm right back outside the room I had just gone into. I have never had a game scare me quite like that...or really mindfsck me that much either.
So, in a sense...Eternal Darkness is the scariest game I've ever played