Scandal: Do Public Servants Have the Right to Private Lives?

Discussion about current scandals, such as General Petraeus who recently resigned after the exposure of his extra marital affair, and former congressman Mel Reynolds, who resigned after allegations he had sex with a 16 year old campaign worker and recently announced he is running again for Congress. Should the sex lives of public officials determine their fitness for office? Is an extra marital affair a true compass for ones moral character, and ethics on governmental issues? Or are we being hypocritical - (let he who is without sin...etc.)? Should officials be judged by their public works and not their private lives?