1. Slice kumquats into 1/4-inch-thick pieces; remove seeds. Combine kumquats and granulated sugar in medium saucepan. Cook slowly over low heat, stirring often with wooden spoon, until kumquats start to release their juices, 10 to 15 minutes. Continue to cook over low heat, stirring often, until kumquats have softened and liquid has thickened slightly, 25 to 35 minutes. Transfer to small bowl.

2. Combine apricots, brandy, 1/2 cup water, ginger, and zest of 1 lemon in medium saucepan. Cover; bring to boil over medium-high heat. Reduce to simmer, and cook until almost all liquid has evaporated, 15 to 20 minutes. Remove ginger; discard. Transfer to food processor; puree until smooth; set aside.

3. Butter an 8-to-10-cup pudding mold and a circle of parchment paper that is 4 inches wider in diameter than the mold.

4. In electric mixer, cream butter and brown sugar on medium speed until lightened, 1 to 2 minutes. Beat in eggs, one at time, until incorporated, 1 to 2 minutes with each addition. Add vanilla and remaining zest. Slowly beat in apricot puree. Sift flour, baking powder, and salt; add to apricot mixture in two additions, alternating with milk.

5. Spoon about 3/4 cup kumquat marmalade into mold, arranging kumquats around sides of mold to follow pattern if there is one; reserve remaining marmalade. Pour in batter. Tap mold sharply down on counter to distribute batter evenly. Cover with parchment, and secure with rubber band; cover with foil. Place lid over foil onto mold. Place rack in stockpot, and set mold, lid side up, on rack. Pour in boiling water to halfway up side of mold. Cover, and bring to boil over high heat. Reduce to medium low, and steam 2 hours and 20 minutes. Remove; let sit 15 minutes. Uncover, invert mold onto a serving plate, and garnish with sliced kumquats, if desired. Serve pudding warm or at room temperature with dollop of creme fraiche or double cream and reserved marmalade on the side.