Is it safe to replace oleanders with fruit trees?

I have 3 very large oleander along my back wall. The previous owners did not clean around them.. ever. I'd like to replace them with fruit trees. I was told the fruit wouldn't be safe to eat. Is this true?
  2 answers