On the subject of personality factors and success in science, here is a provocative essay by UK professor Bruce Charlton. (PDF version.) He claims that the modern system selects for conscientiousness over raw intelligence, with negative consequences.

Question: why are so many leading modern scientists so dull and lacking in scientific ambition?

Answer: because the science selection process ruthlessly weeds-out interesting and imaginative people. At each level in education, training and career progression there is a tendency to exclude smart and creative people by preferring Conscientious and Agreeable people. The progressive lengthening of scientific training and the reduced independence of career scientists have tended to deter vocational ‘revolutionary’ scientists in favour of industrious and socially adept individuals better suited to incremental ‘normal’ science. High general intelligence (IQ) is required for revolutionary science. But educational attainment depends on a combination of intelligence and the personality trait of Conscientiousness; and these attributes do not correlate closely.

...At each level in education, training and career progression there is a tendency to exclude smart and creative people by preferring conscientious and sociable people. As science becomes ever-more dominated by ‘peer review’ mechanisms, pro-social behaviour in scientists has been accorded primacy over the brilliant and inspired – but abrasive and rebellious – type of truth-seekers who used to be common among the best scientists.

A majority of senior professional scientists have been through a rigorous and prolonged process of education, selection and training to become professional researchers. Yet the nature of the rigour and the duration of the process in modern science ensures that those who come out at the end and attain long-term scientific employment are not the kind of people capable of top level, revolutionary science. They will very probably be extremely productive and socially compliant, but of only moderately high intelligence and likely to be lacking in imagination [2].

...Modern science is just too dull an activity to attract, retain or promote many of the most intelligent and creative people. In particular the requirement for around 10, 15, even 20 years of postgraduate ‘training’ before even having a chance at doing some independent research of one’s own choosing, is enough to deter almost anyone with a spark of vitality or self-respect; and utterly exclude anyone with an urgent sense of vocation for creative endeavour. Even after a decade or two of ‘training’ the most likely scientific prospect is that of researching a topic determined by the availability of funding rather than scientific importance, or else functioning as a cog in someone else’s research machine. Either way, the scientist will be working on somebody else’s problem – not his own. Why would any serious intellectual wish to aim for such a career? ...

Shorter Charlton: there are too many hoops, and we end up selecting for Agreeableness and Conscientiousness (hoop jumping abilities) rather than raw brainpower.

I partially agree with Charlton's claims, but the specifics vary from field to field. The area he seems most familiar with is medical science, which most physicists (after teaching premeds and biology students) might concur selects for conscientious rather than brilliant types ;-) In physics it seems we are quite tolerant of odd personalities -- hyper aggressive types, those with Asperger's Syndrome, etc., especially if the person in question displays tremendous ability. I would guess the same is largely true in math and engineering. In biology and medicine it may not be that easy to tell the really talented researchers from the rest (at least at early career stages), which would lead to more emphasis on personality traits. It's also true that in many areas of physics (specifically, but not limited to, the theoretical ones) one can work as a single investigator or small group lead investigator quite early. This may be less true in medicine and biology.

I discussed the current incentive system in science here, as well as the job prospects in theoretical physics. Given the situation I can't blame any students who find that alternative careers might be preferable. As I wrote here (in partial agreement with Charlton), this leads to a different kind of selection than in the past:

...Nowadays, success in science seems to be as much a selection for [certain] character or personality traits as it is a selection for talent.

Related posts: frauds , success vs ability .

Regarding Charlton's deeper question: Where have all the geniuses gone? I offer the following from this earlier post. See also Genius, Gleick's biography of Feynman, especially pp.325-328.

... the exact topic discussed in James Gleick's book Genius. In a field where sampling of talents is sparse [like science in its earlier days] ... you might find one giant ... towering above the others, able to do things others cannot. In a well-developed, highly competitive field like modern mathematics, all the top players are "geniuses" in some sense (rare talents, one in a million), even though they don't stand out very much from each other. In Gleick's book, Feynman, discussing how long it might have taken to develop general relativity had Einstein not done it, says "We are not that much smarter than each other"!

To put it simply, if I sample sparsely from a Gaussian distribution, I might find a super-outlier in the resulting set. If I sample densely and have a high minimum cutoff for acceptable points, I will end up with a set entirely composed of outliers, but who do not stand out much from each other. Every guard in the NBA is an athletic freak of nature [and they would destroy their predecessors from the early era of professional basketball], even though they are evenly matched when playing against each other.