Arrange 4 plum quarters on each of 8 skewers so that cut sides of fruit all face in the same direction. Brush cut side of fruit with oil, and place, cut side down, on a medium-hot grill, off direct heat.
Grill just until fruit begins to turn golden brown, about 2 minutes. Brush bottoms of fruit with oil, and turn over. Grill until pit cavity fills with juices and fruit is soft, about 5 minutes, depending on ripeness of fruit and heat of grill.
Serve with bowls of vanilla ice cream, drizzled with caramel sauce.