I'm hoping someone can help me figure out where I went wrong...
I added the PWM lines to the config.txt. I then wired the new L and R out to the L and R in on my PAM8403.
I then wired the amp L and R +'s to the pot ins. Then I wired the L and R pot outs to the headphone left and right in.
I wired the pot ground to the headphone ground and headphone ground to the power strip ground. The PAM8403 is powered and grounded to the power strip as well.
I then proceeded to generate smoke from (almost positive) the potentiometer. I cut the audio from Pi to test that nothing was fried on the Pi. Lucked out there.

I used the pins from wermy's vid for the pot and the headphone diagram from Felder.
I didn't put my low pass filter together yet, but I can't imagine that would have helped. Did I mess up some pins somewhere?
Pardon my rats nest. It will all look about 30% more organized when I start putting it in the case.

Thanks all!