I've had the same problem on my 2004 Forester XT. The best explanation / fix I've found is the magnetic clutch gap has worn a bit too much and can't stay pulled in. Adjusting the gap involves pulling the clutch plate hub and removing a shim to reduce the gap distance. I haven't tried to fix mine yet since I just rebuilt the engine after a spun rod bearing "event".
I'll try to find the link to post here. http://www.subaruforester.org/vbulletin/f87/how-adjust-c-compressor-clutch-116835
This is a different post than the one I had found before, but uses similar procedure and tools to do the adjustment.