From the source.

You can get flippin’ fresh fish direct from the fishers if you are visiting regional Victoria. Pack an Esky, they’ll provide the ice, and head to a fishing co-operative. These are bodies formed by commercial fishers to market and distribute their catch. You can buy fresh fish from the co-op stores in Apollo Bay, Lakes Entrance and San Remo. The Apollo Bay and San Remo co-ops also sell cooked fish and chips while at Lakes Entrance there is a viewing platform from which to watch the fishers unload their catch.

Special government licences allow some fishers to sell their catch direct from the boat. Look for mussels sold off the boat at Flinders pier and Portarlington pier. Seasonally fish is available from small fishers in the fishing ports. Some aquaculture businesses have farm gates where you can buy freshly processed products from barramundi to smoked trout.