Preheat oven to 375 degrees. In a food processor, pulse ginger until finely chopped. Add brown sugar, allspice, and bread. Pulse until coarse crumbs form.
Sprinkle 1/2 cup of crumb mixture in the bottom of an 8-inch square (2-quart) baking dish; fill with mangoes, and drizzle with lemon juice. Level fruit with the back of a spoon.
Top with remaining crumb mixture; dot with butter. Bake until crumbs are golden brown and fruit is bubbly, about 45 minutes. Serve.