After trying multiple things on Buster, I abandoned that approach and followed your path. The numpy distribution requirement for the install is not compatible with Python3.7 and I didn't want to get involved in manual upgrades to 3.9.
With Bullseye, I encountered the same numpy install issues...