In a bowl, whisk together orange zest and juice, vinegar, honey, and 1/4 teaspoon pepper. Place pork, orange wedges, and rosemary in a shallow dish or zip-top bag and coat with marinade. Refrigerate pork at least two hours or up to overnight.
Remove pork, orange wedges, and rosemary from marinade; transfer pork and oranges to a wire rack set on a foil-lined rimmed baking sheet. Season pork with salt and pepper. Transfer marinade to a small saucepan, simmer over medium until reduced by half, about 5 minutes.
Heat broiler, with rack in upper third of oven. Broil pork and oranges 3 minutes, then flip pork, brush with reduced marinade, and add rosemary. Broil until pork is lightly charred and cooked through, 5 to 7 minutes. Brush with more reduced marinade and serve.