Maybe I'm missing something.
Making a starter isn't all that hard , is it?
Pour in some DME and boil it and let it cool?
You need to wash the slurry, and let it settle.
It may be faster, but there is more risk as well.
There is greater chance of infection, and of mutation.
You don't need to wash slurry. You might not want to continue to re-use slurry ad-infinitium, but a few consecutive brews is OK and you will not notice any change unless you have really abused the yeast. Even slurry from a stout into a pale beer is not going to make any noticeable change to the beer. If you practice reasonable sanitation, you will not have any infection issues - I haven't and have been doing this for years.
Let settle a few days, pour off the liquid, and what you have is trub (that will fall to the bottom), some dead yeast (which is actually yeast nutrient), and healthy yeast
Using about a quarter of the yeast is about the right pitch if used reasonable quickly, and will reproduce plenty of healthy new yeast in the new beer.
So no starter, so a lot less effort than messing with a new starter. Starters are not hard, but are extra time and effort, and you have to make sure you have it ready for brew day.
Some will say to use slurry within a week, I will straight pitch it up to a month in the fridge.
As I said, you can't continue this forever. The yeast will eventually start showing signs of mutation. But for a few (to maybe 5) brews, go for it.
Like you I take a small amount from my starters and store them for future use, but I'm keeping mine for 18 to 24 months before I wake them up again, as I have several strains going at once, and rotate thru several others.
With respect to potential infection. I'm not sure, but there might be more opportunity with making multiple starters than using slurry. Every time you handle it you have opportunity for infection, and Ithink you handle it more by stepping up starters.