Place granulated sugar and 1/3 cup water in a small saucepan; set over medium heat. Bring to a boil, stirring occasionally. When sugar has dissolved, remove from heat; add amaretto, if using, and almond extract. Let syrup cool.
Using a serrated knife, slice the two cake layers in half horizontally, forming four layers. Crumble one into the bowl of a food processor; process until fine crumbs form. Set aside.
Generously brush tops of remaining three cake layers with cooled syrup. Place one cake layer on a 6-inch cardboard cake round.
Place ice cream in the chilled bowl of an electric mixer fitted with paddle attachment. Beat on medium-low speed until spreadable, about 2 minutes. Spread 2 cups ice cream over first cake layer. Crumble 1/2 cup almond crunch over ice cream. Top with a second cake layer. Transfer cake and ice cream to freezer until firm, about 10 minutes.
Remove cake and ice cream from freezer. Spread 2 cups ice cream over top. Crumble 1/2 cup almond crunch over ice cream. Place third cake layer on top. Spread 2 cups ice cream over cake layer. Transfer cake and ice cream to freezer for 30 minutes.
Remove cake and ice cream from freezer; stir remaining ice cream with rubber spatula to resoften. Using an icing spatula, quickly spread remaining 2 cups ice cream around sides of cake. Return cake to freezer for 30 minutes.
In a chilled bowl, whip cream and confectioners' sugar with a hand mixer until soft peaks form. Remove cake from freezer; frost top and sides of cake with whipped cream. Pat reserved crumbs onto sides of cake, and crumble remaining cup almond crunch over the top. Return cake to freezer; freeze until completely hardened, at least 2 hours or overnight.