This sherry- and brandy-infused trifle was inspired by a Victorian recipe.
Put yolks and sugar in the bowl of an electric mixer fitted with the whisk attachment; beat on medium-high speed until mixture is pale and thick, 2 to 3 minutes. On low speed, beat in flour and salt. Gradually beat in half of warm milk.
Pour milk-yolk mixture into remaining milk in pan. Bring to a boil over medium heat, whisking. Cook, stirring, until thickened, about 2 minutes. Pour through a fine sieve into a bowl.
Transfer to an ice-water bath; let cool, stirring occasionally. Stir in vanilla and brandy. Place plastic wrap on surface. Refrigerate until ready to use, up to 3 days.
Beat cream just until stiff peaks form. Fold into pastry cream. Place 7 ladyfinger halves at bottom of a 12-cup trifle dish; spread with one quarter of jam. Pour in one quarter of sherry. Spoon one quarter pastry cream on top. Sprinkle with one-third each almonds and amaretti. Repeat layering 3 more times (last layer does not need almonds and amaretti). Refrigerate 3 hours or overnight.