I say keep the numbers-matching slant-6 and transmission on a pallet in a corner of your garage, packed with oil/grease for long-term storage, do the same for the rear-end (it will not withstand serious V8 power), get a set of conversion engine mounts from Schumacher, a built small-(or big-)block and transmission, electronic ignition, a 8 3/4 rear, new driveshaft and hardware, torsion bars, etc..
It's all a question of money (and storage space), but since you are at least considering the collector value of your car, don't do any mods that can't be undone.
This way, you can have fun with your Challenger, ans still return it to its original condition in 2 week-ends time if you want to.
You could also build the slant-6, but performance slant-6 parts are rare as hen's teeth (read : it will cost more to get 200HP out of a slant-6 than 400 out of a 318), and if something goes wrong, the broken rod will go through your matching numbers block, not some boneyard small-block that can be replaced.
Take your time, look at the possible options, check how much it's going to cost you, and be realistic about what you want to do with the car : Right now, it's a nice cruiser. You can turn it into a very nice cruiser and an amusing strip car with no chassis modifications. You can't get very much under 12-13 seconds without serious mods.
There were 9,929 hardtop Challengers built in 1970 with the slant-6.
So I guess the collector value of this mint slant-6 car will be nowhere near that of the same car with R/T goodies and a big-block (at least during my lifespan). If it was mine, I'd drop a stroked big-block inside it, install torque boxes, frame connectors,and a strong rear-end.
Your car, your money, your decision. And your sleepless nights with a Summit Racing catalog...