Yeah, you can pull if out after the hot break, or just boil it for a bit in a separate, smaller pot.
You can always to a mini-mash of grain to make a batch of starter wort before you do your real mash. It's maybe a bit time consuming, but it means you won't have to wait a whole day or two from the time your wort is done boiling to when you pitch your yeast. That's such a danger-zone for infection.
Just get a grain bag, fill it with a pound of 2-row. Mash it in a small pot with 1.5 liters of water in it at 152f for an hour. Then rinse it with 2 liters of 170f water. Then boil it, cool it, and pitch the yeast in. Starter's done, beers not getting infected.
Or make a larger amount of starter wort and freeze it for next time. You can even make starter wort using rice along with 2-row if you don't have lots of spare grain.