Set a large fine-mesh sieve over a large bowl. Place bottom of bowl in an ice bath; set aside. Off heat, in a medium saucepan, whisk together sugar, cornstarch, nutmeg, and pumpkin-pie spice. Gradually whisk in 4 cups milk, taking care to incorporate the cornstarch. Whisk in egg yolks.
Whisking constantly, cook over medium heat until the first large bubble sputters, 10 to 12 minutes. Reduce heat to low; cook, whisking constantly, 1 minute more. Remove from heat; immediately pour through sieve into bowl. Stir in remaining 4 cups milk. Let cool completely in bowl still set in ice bath.
Just before serving, stir in brandy, and ladle into glasses; garnish with pumpkin-pie spice.