I'm not sure if I believe this, mainly because I know of no evidence that gay people actually pay more attention to their relatives who are not their children that straight people do. (Of course, just because I haven't heard it doesn't mean it's not there.)
Furthermore, on average people share half their genes with their children and one-quarter of them with a niece or nephew. So in order for this to work out in some sort of "expected value" framework, a gay person would have to be able to enhance the survival probability (or, more accurately, the expected number of children, or grandchildren) of their nephews and nieces by twice as much as they'd help their children, if they had them.
However, this could have an effect in times when "expected value" isn't what really matters -- when a family (and therefore a set of people with similar genes) are just barely clinging to life, very close to dying out. The logic then is that a straight person and their gay siblings can put all their eggs in one basket -- and then watch that basket very carefully.
Although I've never heard this sort of argument applied to homosexuality, I have heard it applied to various mental illnesses. There are people who believe that although, say, schizophrenia is obviously very harmful to the people who suffer from it, certain good qualities (say, high intelligence -- I don't remember if this is actually one of them!) tend to occur in the near relatives of schizophrenics. (Let me just say that I in no way am attempting to compare homosexuality and schizophrenia.)
Let's say, hypothetically, that there are ten genes, each of which occur in two variants called "red" and "blue", which cause schizophrenia. Let's say that each of these is "red" with probability p and "blue" with probability q = 1-p. Furthermore, a person which has zero or one of the "red" genes is "normal"; one who has two is of high intelligence, or "smart"; one who has three of more is schizophrenic. Assuming people mate at random, we are in a state of Hardy-Weinberg equilibrium. We compute:
- The probability of a person having zero or one "red" genes is P1 = q10 + 10q9p.
- The probability of a person having two "red" genes is P2 = 45q8p2.
- The probability of a person having three or more "red" genes is P3 = 1 - (P1 + P2). (It can be written out as the sum of the probabilities of having 3, 4, ..., 10 red genes, but it's easier to compute this way.)
What we see here is clear. When the frequency of red genes is low, most people are normal. When the frequency is moderate, we see a large minority of smart people and a small minority of schizophrenics. When the frequency is high, the schizophrenics begin to outnumber everybody else. Presumably, then, evolution would want (and here I commit the common sin of anthropomorphizing evolution) a moderate frequency of the "red" genes. As for how that is created, I think that assuming that high intelligence has survival value will do it; when the red genes are rare, "normal" people are the most common but the smart people will out-reproduce them, increasing the frequency of red genes; and when the red genes are common, schizophrenics are the most common but the smart people will out-reproduce them, decreasing the frequency of red genes. But even at the equilibrium point, not everyone will have exactly two red genes, which is what you need to be smart in this model. So there will still be variation.
Something similar actually goes on with the inheritance of sickle-cell anemia; having two copies of a certain allele gives people sickle-cell anemia, but having one copy of that allele confers resistance to malaria.