YouTube will no longer allow videos that "maliciously insult someone" based on "protected attributes" such as race, gender identity or sexuality.
The video-sharing platform will also ban "implied threats of violence" as part of its new harassment policy.
A row erupted in June after a prominent video-maker said he had been the target of abuse by another YouTube star.
At the time, YouTube said its rules had not been broken. But it has now deleted many of the videos in question.
"Even if a single video doesn't cross the line, with our new harassment policy we can take a pattern of behaviour into account for enforcement," Neal Mohan, chief product officer at YouTube, told the BBC.
As a result of the policy change, the Google-owned business also considered taking down clips of President Trump calling Democratic Senator Elizabeth Warren "Pocahontas" to taunt her over her claim that she has distant Native-American heritage.
But it decided that the president's intention was to curry favour with voters rather than attack his opponent on race and so the videos could remain online.
What was the row about?
At the centre of the dispute was journalist Carlos Maza, who presented videos for the Vox channel.
Mr Maza said he had been the subject of persistent abuse from rival video-maker Steven Crowder, who presents a talk show on YouTube.
Mr Crowder had about four million subscribers at the time, while Vox had about six million followers.
Mr Maza created a video compilation of all the times Mr Crowder had made fun of his sexual orientation and ethnicity.
In the clips, Mr Crowder imitated Mr Maza's accent and called him, among other things, a "lispy queer", a "gay Vox sprite" and a "gay Mexican".
Mr Crowder said the language he had used to mock Maza was "friendly ribbing".
But the videos led many of Mr Crowder's viewers to harass Mr Maza on social media.
YouTube said it had conducted an "in-depth review" and found "the videos as posted don't violate our policies".
However, it later restricted Mr Crowder's ability to earn advertising revenue due to "continued egregious actions".
On Wednesday, it was expected YouTube would delete several of Mr Crowder's videos that fell foul of its updated harassment policy. But it appears he made the material private in advance.
What does the updated policy say?
Prior to the change, YouTube had already banned videos that:
contained explicit threats of violence
bullied somebody about their appearance
revealed somebody's personal information
encouraged viewers to harass an individual
But the new policy also bans:
"veiled" or implied threats of violence, such as saying "you better watch out"
simulated violence towards an individual
malicious insults based on protected attributes such as race, gender expression or sexual orientation
YouTube said the new policy would apply to "everyone" including politicians and popular YouTube stars as well as the general public.
Video-makers who consistently break the rules will have their ability to earn advertising revenue restricted, and may have videos deleted or their channel closed.
The company said there would be some exemptions from the new policy, including insults used in "scripted satire, stand-up comedy, or music".
Mr Mohan told the BBC that individual complaints would have to be judged on a case-by-case basis, with the context of each video being taken into account.
However, the new guidelines do say: "This is not a free pass to harass someone and claim, 'I was joking.'"
Mr Mohan said YouTube had consulted with think tanks, video-makers, Google employees and other third parties to help inform its policy.
He told the BBC he did not think the new rules would restrict YouTubers' freedom of expression.
"We don't want YouTube to be a place where public discourse is getting stifled as a result of people having the fear of being harassed on our platform," he told the BBC.
"My view is that, on balance, having a strong framework around which we are protecting individuals from being harassed is important to ensure that our platform remains one where there can be robust debate."
Source : BBC