There were several critics who took issue with the fact that the story wasn’t a true depiction of what really happened in the town. While some of those people likely had a problem with racism being so broadly displayed on screen, there were those with more personal grips.
If you’ve seen the movie, you probably remember the scene where Jon Voight is having sex with his store clerk Jewel, played by Akosua Busia. Jewel was based on a real woman and this real woman had several children who were still living when the movie was released. One of her sons was not too happy about his mother being portrayed as a promiscuous woman. Her son was so upset about the portrayal that, legend has it that he had a heart attack and died six months later.