Make the peperonata: In a large skillet, heat oil over medium heat; add onion and garlic, and cook, stirring, until onion is translucent, about 3 minutes. Add remaining ingredients, stirring to break up anchovies. Season with salt and pepper. Reduce the heat to a simmer, and cook, stirring occasionally, until peppers are very soft, about 45 minutes.
Preheat oven to 500 degrees. Make the bass: Place 2 lemon slices on a small rimmed baking sheet; top with a piece of bass. Repeat process with remaining lemon slices and bass. Season with salt and pepper. Drizzle evenly with oil, and top each piece of fish with a sprig of rosemary. Roast until the fish is cooked through, 10 to 12 minutes.
Serve fish over peperonata with potatoes on the side.