Preheat oven to 450 degrees. Arrange plums, cut side down, in a baking dish. Season with salt, pepper, and sugar, if desired; set aside. Score skin of duck breasts in a crosshatch pattern, about 1 1/2 inches apart. Season with salt and pepper.
Place breasts skin-side down in a cold medium skillet. Place over medium-high heat until most of the fat has rendered and the skin is dark golden brown, 5 to 8 minutes. Pour off fat, and turn duck breasts, meat-side down.
Place duck and plums in oven. Roast until meat thermometer inserted into the thickest part of the duck registers 125 degrees, about 10 minutes. Remove duck and plums from oven. Remove duck from skillet, and cover to keep warm. Let rest for 10 minutes.
Slice each breast half into 12 thin slices. Place 6 slices on each plate. Garnish each plate with 2 plum halves. Drizzle with any accumulated juices from the bottom of the baking dish.