If you plant trees adapted to your climate and learn the basics of care, fruit trees are not especially difficult to grow. Fruit trees do require regular maintenance, however, and can become problematic after a period of neglect. The degree of difficulty is mostly determined by the grower.
Fruit trees kept small by summer pruning are much easier to care for than large trees. Small trees are much easier to prune, thin, spray, protect and harvest. For more about keeping backyard fruit trees small, see Backyard Orchard Culture.
An equally relevant question is "Are home fruit trees worth the effort?" If planned and managed properly, the benefits of fruit trees can far outweigh the effort. For some of the many reasons to grow fruit at home, see Introduction to Home Fruit Growing.