For the cake: Preheat oven to 400. Butter three 8-by-2-inch round cake pans. Line bottoms with parchment cut to fit, and butter parchment.
Whisk together flours, confectioners' sugar, nutmeg, and salt in a bowl.
Finely chop fruits and peels; coarsely chop almonds. Mix with brandy.
In the bowl of a standing mixer fitted with the whisk attachment, beat egg whites on medium speed until foamy. Gradually beat in granulated sugar. Raise speed to medium-high, beating until stiff peaks form. Transfer to a large bowl. Sprinkle a third of the flour mixture over the peaks, and fold with a spatula until just combined. Gently fold in another third of the flour mixture. Toss brandied fruits and nuts with remaining flour mixture, then fold into batter.
Divide batter evenly among prepared pans, and smooth tops. Bake until golden brown and cakes pull away slightly from sides of pans, 15 to 17 minutes. Let cool in pans on wire racks for 20 minutes. Run a knife around edges to loosen, then invert cakes onto racks to cool completely. (Cakes can be wrapped in plastic and refrigerated overnight.)
For the frosting: Beat cream cheese with a standing mixer on medium-high speed until soft, 3 to 4 minutes. Reduce speed to low, and beat in confectioners' sugar. Raise speed to medium-high, and beat until fluffy, about 3 minutes. Switch to the whisk attachment. With mixer on low speed, beat in 1 1/2 cups cream and the vanilla. Add creme fraiche, and beat until smooth, scraping sides of bowl as needed. (Frosting is best used the day you make it; refrigerate it in an airtight container until needed.)
To assemble: Place 4 1/2 cups frosting in a large bowl, refrigerating remainder for glaze. Whisk until very stiff peaks form. Place 1 cake on an inverted baking sheet lined with parchment. Brush with 2 tablespoons jam. Using an offset spatula, spread 3/4 cup frosting over jam. Repeat with second cake, 2 tablespoons jam, and 3/4 cup frosting. Top with remaining cake. Refrigerate remaining 3 cups frosting. Freeze cake for 1 hour.
Spread remaining 3 cups frosting over top and sides of cake, smoothing with an offset spatula. Freeze for 1 hour.
For the glaze: Add remaining 1 1/2 cups cream to the refrigerated frosting, and whisk until combined. Place cake on a wire rack set on a parchment-lined baking sheet. Using a ladle, spoon glaze over cake, allowing it to drip down sides, until cake is covered. (You will have a generous amount of leftover glaze. The glaze on the cake will appear slightly translucent but will become more opaque when chilled.) Refrigerate, uncovered, at least 4 hours or overnight.
Using 2 spatulas, transfer cake to a serving platter or a cake stand. Sprinkle sanding sugar over top, and gently toss it onto sides to coat entire cake. Wipe sugar from edges of platter. Garnish with sugared lemon zest if desired. Serve immediately. (Cake will keep at room temperature for about an hour but is best-and easiest to slicewhen cold.)