You can only exhaust Atk/Def dice that you have access to - you cannot exhaust more than your respective Atk/Def stats using this Backup Plan. So if your Def stat is 2, you cannot use this skill to exhaust a 3rd or 4th Def die. If you have a temporary boost to one of your stats (for example, if you've used Perseverance to boost your Atk stat for the turn), you can then use Vanish to exhaust those temporary Atk dice.
As normal, exhausted dice reduce the number of those dice that you can roll on your turn.
The supply of Atk/Def dice is meant to be considered unlimited. The reason this skill tells you to take the dice from the supply is simply to clarify that you don't have to remove Atk/Def dice from anywhere on your Gearloc mat when you use this Backup Plan (except, of course, whatever bones you remove from your BP to trigger this ability).