1. To clarify butter, place in a small saucepan, and melt over medium heat until bubbling. Remove pan from heat, and cool, 2 to 3 minutes. Using a spoon, carefully skim the milky foam from the top of the butter, and discard. Carefully pour the skimmed butter into another bowl, leaving behind the milk solids and discarding; set clarified butter aside.

2. To make syrup, combine 1/2 cup sugar and 1/2 cup water in a small saucepan. Stir in honey. Bring to a boil, and stir until sugar is dissolved. Remove from heat, and stir in brandy, lemon juice, orange juice, and zest; set aside to cool.

3. Heat oven to 350 degrees. Lightly butter an 8-by-8-inch Pyrex or metal baking dish. Heat milk in a medium saucepan until hot but not boiling. Add lavender flowers, cover, and let steep about 20 minutes. Strain and return milk to saucepan. Whisk remaining 1/2 cup sugar and semolina into milk. Place over medium-low heat, and cook, stirring constantly with a wooden spoon, until thick and creamy, 8 to 10 minutes. Remove from heat, and cool slightly.

4. Beat the eggs by hand, or in an electric mixer until pale and creamy, about 2 to 3 minutes. Quickly pour the eggs into the semolina mixture, stirring constantly. Let mixture cool before assembling pastry.

5. Layer 8 sheets of phyllo pastry in baking dish, brushing with clarified butter between each layer and overhanging the edge of the baking dish. Spread semolina mixture over the phyllo, and top with 6 more sheets of phyllo, brushing with butter between each sheet. Fold over the edges of phyllo, and brush top with butter. Cut 2 sheets of phyllo into shreds and toss in a bowl with some of the butter until coated. Sprinkle over top of pastry. Bake until phyllo is golden brown and filling is set, 50 minutes to 1 hour. Remove from oven, and immediately pour 1/4 cup of cooled syrup over the top, making sure it is evenly distributed. Let cool, and serve, if desired, with remaining syrup.