First of all, according to the Oxford dictionary, an olive, just like a plum or a cherry, is a fruit. The olive is the small, bitter-tasting fruit of the olive tree, Olea europea. Olives are classified as fruit because they’re formed from the ovary of the olive flower, and they’re seed-bearing structures—the pits could grow into trees if you planted them. In culinary terms, however, olives are like those other fruit, tomatoes, and are treated as a vegetable.
Green olives are not from a different tree than black olives, they are just picked before they are ripe.