The family is claiming harrassment. The neighbors disagree:
"This letter is not written as a means to harass the family or to begin a witch hunt," the flier said. "We wish the family no harm. Our goal is to peacefully communicate that this kind of hate and ignorance will not be accepted here in our neighborhood where we live and raise our families."
As Professor Volokh noted, since PB specifically decided to move to this town because the old one "wasn't White enough", it is perfectly fair for the neighbors to push back against being identified with that sort of racist sentiment.
What would really be great is if the neighborhood began to actively work to end the racial monotony that attracted PB to the area in the first place. But that might be too much to ask.