Grease a baking pan with butter.
Tear up the brioche and place it tightly all around the pan.
Toss the apples with ½ teaspoon ground cinnamon, brown sugar and the lemon juice.
Add this to the baking pan and tuck some of the apples in between the brioche.
In a mixing bowl, make the custard by whisking the eggs. Add the milk, maple syrup, ½ teaspoon ground cinnamon, allspice and vanilla. Mix to combine.
Pour this custard over the brioche and apples. Press down the brioche to make sure it is coated well with the custard.
Cover and refrigerate till ready to bake.
Preheat oven to 350F.
Place the casserole in the oven and bake uncovered for 25 mins or so, till the custard is set with the brioche to make the baked French toast.
Serve warm with more maple syrup or top with some powdered sugar.