Republican presidential candidate Herman Cain may have kept the spotlight off his sexual assault allegations at the Republican presidential primary debate in Michigan on Nov. 9, but that may not be enough to dispel the dark cloud looming over his campaign, voters said in a new poll.

CBS reports:

“In a new CBS News Poll, 61 percent of Republican primary voters say the sexual harassment accusations against Cain won't make any difference in their vote, but 30 percent say the charges make them less likely to back him, and that rises to 38 percent among women. Cain has lost support among women since last month — from 28 percent in October to 15 percent now.”

The loss of the female vote couldn’t come at a worse time for Cain, who was recently slammed for making an insensitive remark about Anita Hill, the woman who accused Supreme Court Justice Clarence Thomas of sexual harassment 20 years ago. Cain was again in hot water after he referred to former House Speaker Nancy Pelosi as “Princess Nancy” at the Nov. 9 debate, a comment which he later took back.

The CBS poll still puts Cain in the lead among Republican primary voters, with 18 percent of the vote, followed by Mitt Romney and Newt Gingrich, who are tied for second place with 15 percent. Rick Perry, who has been the subject ridicule after he flubbed his words during the most recent Republican debate, came in third with eight percent.

The GOP candidates will debate national security and foreign policy on Saturday night in Spartanburg, South Carolina. The debate airs at 8 p.m. ET on CBS.