Preheat oven to 375°F (190°C) and grease a standard 12-cup muffin tin.
Whisk together flour, granulated sugar, baking powder, and salt in a large mixing bowl.
In a separate medium bowl, combine milk, melted butter, egg, and vanilla extract.
Fold the wet ingredients into the dry ingredients until just moistened; do not overmix.
In a small bowl, combine the softened butter, brown sugar, and cinnamon to form a thick paste.
Fill each muffin cup halfway with batter, then add a small teaspoon of the cinnamon paste.
Cover with the remaining batter and dollop the remaining cinnamon paste on top, swirling lightly with a toothpick.
Bake for 18 to 20 minutes until the muffins are golden and a tester comes out clean.
Whisk powdered sugar and milk in a small bowl until smooth to create the glaze.
Drizzle the glaze over the muffins while they are still warm.