This is scale! You can spray with insecticidal soap or Horticultural oil. Now will be a good time because the "crawlers" are crawling now. Here is an insecticide that will help. Imidacloprid (Bayer Advanced Garden Tree & Shrub Insect Control Concentrate) is applied as a drench around the root zone of infested plants. This water soluble insecticide is taken up by the roots and transported throughout the plant where it is ingested by sap feeding insects. This provides a means of scale control without reliance on sprays. However, it may need to be applied several weeks before crawlers are active for best results.
the little black things are scale and the buildup on the trunk as well?
You can also use Neem Oil.
Scale can be a pain to get rid of, but I would caution about using a systemic (or any chemical that is absorbed by the roots if you plan to eat the fruit! I'd try Neem first and maybe insecticidal soap if it were my tree.
Neem oil and gentle scrapping with your gloves on.