Heat oven to 350 degrees. Generously butter a 2 1/2-quart casserole dish, and set dish aside. Snap off and discard tough ends of asparagus. Using a vegetable peeler, carefully peel outer layer from bottom 2 inches of asparagus spears.
Fill a large straight-sided saute pan with salted water, and bring to a boil; prepare an ice bath. Add asparagus, and cook until bright green, about 1 minute. Transfer asparagus to ice bath, and let cool. Drain, transfer to a clean kitchen towel, and pat dry. Set asparagus aside.
Melt 1/2 tablespoon butter with 1/2 tablespoon olive oil in a medium skillet over medium heat. Add mushrooms, and cook, stirring once or twice, until mushrooms turn golden brown, 1 to 2 minutes. Transfer mushrooms to a plate; set aside.
Melt remaining tablespoon butter with remaining tablespoon oil, and add onion. Cook until onion is soft and translucent, about 15 minutes. Remove the pan from heat, and set aside.
Randomly arrange half of the bread pieces on bottom of prepared casserole dish. Sprinkle half of the rosemary over the top. Place half of the asparagus, onions, mushrooms, and cheese on top. Make a second layer with remaining bread, rosemary, asparagus, onions, mushrooms, and cheese.
To make custard, combine milk and cream in a medium saucepan, and set over medium-high heat; cook until mixture is hot. In a medium bowl, whisk together eggs, hot-milk mixture, 1 1/2 teaspoons salt, and 1/4 teaspoon pepper until combined.
Pour custard over pudding. Press down on pudding, completely soaking top layer of bread, and place a plate on top. Weigh down the plate with a can or other object. Let pudding stand 20 minutes.
Bake pudding until custard has set and top is golden, about 45 minutes. Remove dish from oven, and transfer to a wire rack to cool, 5 to 10 minutes. Serve.