In a medium saucepan, combine olives, olive oil, orange slices, lemon slices, and fresh thyme. Season to taste with red-pepper flakes. Bring to a simmer and cook, stirring occasionally, 5 minutes. With a slotted spoon, transfer olives, fruit, and herbs to a serving dish; reserve oil for another use.
To store, let cool and refrigerate in an airtight container, up to 1 week.