Not quite right about the PNG there. We're mainly talking about the differences when used on the web. When a GIF or JPG are interlaced, yes, it looks like stripes building up. A fast connection hardly notices it now, while on a slow dial-up connection it would be clearer.If they are not interlaced, then the browser has to wait until the full image has downloaded before displaying it.
The PNG can have interlacing, but the outline and shape of the image start to build up together - as it downloads it makes its required space known to the browser and fills the gap instantly, but (if your connection were slow enough for you to see it) it starts off fuzzy and gradually 'sharpens'. I say 'gradually' but we're talking very small fractions of a second here. A non-interlaced PNG works the same way as a non-interlaced JPG or GIF - it has to download completely before appearing.
In real terms, nowadays, it makes very little difference to most people. Back in 1995 when the 'algorithm' was first created, fast download times were vital - there used to be a phrase that said that if a webpage hadn't loaded in 26 seconds (!) people would just move on to something else.
The main advantage of a PNG now is that it can have transparent areas - so pictures no longer really need to be rectangular (GIFs have been able to have transparent areas, but they use much fewer colours and are not good at detail). PNGs have transparency and are at least as sharp as (but bigger than) JPGs, as the PNG compression is not 'lossy'.
Many people say that we should forget JPG entirely, now that connections are faster. They may have a point.
All the best,
Geoff