The Murdochs are leaning toward not allowing Bill O'Reilly to return to air, according to a report from New York magazine.

According to the magazine's sources, James and Lachlan Murdoch believe that the Fox News commentator, who is currently on vacation, should be fired. Their father, Rupert Murdoch, has pushed against this, saying that it would appear that O'Reilly's firing was because of the New York Times.

Advertisers started dropping O'Reilly's show after allegations that he sexually harassed several women.

While the Murdochs have not yet made a decision, Fox is making plans for its lineup without O'Reilly, NPR reported.

Fox did not immediately return a request for comment from USA TODAY.

The story about the Murdochs potentially dropping O'Reilly broke following protests Tuesday outside of Fox News headquarters in New York. Survivors of sexual assault and harassment, as well as members from the women's advocacy group Ultra violet, called for Fox to fire O'Reilly. Planes flew overhead with banners attached that read, "FOX: #DROPOREILLY, THE SEXUAL PREDATOR."

"For years, Fox News executives have provided cover for Bill O’Reilly’s systematic harassment and abuse of other Fox New’s employees — making it hard to take their promises to investigate such crimes seriously," Shaunna Thomas, co-founder of UltraViolet, said in a statement. "Fox News needs to prove to the public — and its advertisers — that the era of covering up sexual assault is over. That means firing Bill O’Reilly immediately and conducting a comprehensive and independent investigation into sexual abuse at the network."