Make the pastry cream base: whisk the granulated sugar and egg yolks in a medium bowl until the mixture is pale and slightly thickened. Stir in the cornstarch and flour until smooth.
Warm the milk in a small saucepan over medium heat until it just begins to steam and tiny bubbles form at the edge (do not boil). Remove from heat.
Temper the egg mixture: slowly pour about one-third of the hot milk into the yolks while whisking constantly, then pour the tempered yolk mix back into the saucepan with the remaining milk. Return the pan to medium-low heat and cook, whisking continuously, until the mixture thickens and holds a ribbon when the whisk is lifted, about 2–4 minutes.
Off the heat, whisk in the butter, vanilla, and lemon zest until completely incorporated. Press plastic wrap directly onto the surface to prevent a skin from forming and chill until cool, at least 30 minutes.
When the pastry cream is cold, loosen it with a few quick whisks to make it smooth. In a separate bowl, whip 1/2 cup very cold heavy cream to soft peaks, then fold it gently into the pastry cream in three additions using a rubber spatula to keep the texture light.
For the mascarpone chantilly: in a chilled bowl, combine the mascarpone, 1 cup very cold heavy cream, powdered sugar, and vanilla. Whip on medium speed until the mixture reaches a fluffy, pipeable consistency—stop as soon as it moves past soft peaks (about 2–3 minutes) to avoid overbeating.
Cut the angel food cake into roughly 1-inch cubes and set up a 9x13-inch (23x33 cm) baking dish or sheet pan lined with parchment for easy removal.
Assemble the cake: spread half of the cake cubes in an even layer across the bottom of the pan. Spoon about two-thirds of the blueberry filling over the cake, leaving roughly 2 tablespoons aside for the top.
Add the remaining cake cubes in a second layer and press lightly to even them. Smooth the cooled vanilla pastry cream over the cake layer in an even sheet using an offset spatula.
Spread the mascarpone chantilly over the pastry cream. Dollop the reserved blueberry filling on top and use a knife or small spatula to swirl it gently into the cream for marbled ribbons.
Cover and refrigerate for at least 4 hours to firm up (overnight is best for clean slices). Just before serving, sprinkle the toasted almond slivers over the top for crunch.
To serve, cut the set cake into 12 squares and keep refrigerated. Leftovers keep well in an airtight container for up to 2 days.