Yes, especially when premoult is possible
Uneaten live prey should not be left indefinitely. The biggest concern is a feeder remaining in the enclosure when the tarantula begins to moult and cannot defend itself.
Crickets
Crickets are particularly worth removing because they can become opportunistic and chew on a vulnerable spider. If one disappears into substrate, use a trap or remove it when it emerges.
Roaches
Roaches often hide quickly. Crushing the head or using a feeding dish for small specimens can make monitoring easier, especially with slings.
How long to wait
If the spider clearly ignores prey, remove it rather than leaving live feeders loose in the enclosure.
Sealed burrows
Do not open a sealed burrow to offer food. A hidden tarantula can fast safely for long periods, and breaking the retreat may interrupt premoult or moulting.
Clean-up
Remove dead prey remains and boluses when accessible. Good feeding practice and spot cleaning reduce mould and mites without requiring constant enclosure disturbance.