Add sugar, cornstarch, and salt to a large sauce pan and stir together. Add in 1/4 cup of milk and stir together with the sugar mixture until completely combined. Then, slowly pour in the rest of the milk.
Place over medium-low heat to thicken the milk into a pudding. This will take about 12-15 minutes. Do not let the pudding come to a simmer or bubble. If it does, reduce the heat.
Once the mixture is thick like pudding, scoop out 1/3 of the pudding and place into a small bowl with the peanut butter. Stir together until smooth. Set aside. Add chopped semi-sweet chocolate to the remaining pudding and stir over heat until smooth. Remove from heat and add vanilla extract. Set aside to cool to room temperature.
Once the puddings are room temperature, it's time to assemble. Grab 8 popsicle molds and add chocolate pudding to the bottom 1/3 of each one. Scoop in peanut butter pudding up another 1/3 full and then finish with chocolate pudding. It's important to scoop the pudding gently with a spoon instead of pouring into the mold so that the layers don't mix together. Add sticks and then freeze for 8 hours or until solid.
Remove from molds and then enjoy!