I lived in NYC as a kid in the 60's, and I'd see these little vent pipes everywhere,
because they were at a kid's eye-level. Always the strong smell of fuel oil.

I once watched as a kid when the guys were delivering oil through giant hoses, and the oil would blast into the giant tanks in the apartment building's basement. The air displaced from the tank has to go somewhere, so I believe these vent pipes run out of the top of the tanks to let the displaced air out. Often there'd be a spray of old oil on the wall around these vents, apparently in evidence of this. And when oil is consumed during use for heating, the vacuum created by the lowering oil level is probably relieved by these vents as well.