Preheat oven to 350 degrees (325 if you have a dark-coated pan). Prep your 9x9 square pan by lining it with parchment paper with a little hanging over for easy cake removal.
Whisk flour, baking powder, ground nutmeg, and salt in small bowl to combine. Set aside.
Whip butter and sugar together with a stand mixer or a hand mixer until it becomes light and fluffy. If using a stand mixer, use the paddle attachment. Add eggs , one at a time, and vanilla. Mix on low until fully incorporated.
Add 1/3 of flour mixture to the butter mixture and mix until just combined. Add 1/2 of the sour cream to the butter mixture and mix until just combined. Repeat again until flour and milk mixtures are fully incorporated into the butter mixture.
Take a spatula, scrap down the sides of the bowl of the cake batter to make sure you have all the ingredients mixed in. Pour half of the batter into prepared dish and spread into an even layer. Mix the diced pears with apple pie spice (or cinnamon) and remaining ⅓ cup of sugar. Top batter with half of sweetened pears. Spread second half of batter over the top and then sprinkle remaining pears on top. Make topping (see below) and sprinkle over the batter.
Bake the cake in the preheated oven for 50 minutes, until a toothpick comes out clean. If you want a crispier top layer, cook for 55 minutes. Once finished baking, place on rack to cool for 10 minutes while still in cake pan. Then remove from cake pan to finish cooling on the rack. Can be served warm or cooled. Store leftover cake in storage container for up to 3 days.