Of the 43 people on board there were 24 survivors, according to operator Precision Air.
The two pilots initially survived and managed to speak to local officials from the cockpit but the prime minister says they may have since died.
The plane crashed near the shore at the end of the Bukoba airport runway.
Rescuers were able to wade through the water to bring some of those still inside the plane to safety.
Abdul Nuri was at the airport waiting for the return flight to Tanzania’s biggest city, Dar es Salaam, when he saw the plane plunge into Africa’s largest lake.
“We were really shocked. People panicked and some started crying and shouting,” he told the BBC.
“At the arrivals gate people panicked as well – most of them were waiting to welcome their relatives.”
He has spoken to the fishermen who were first on the scene. They told him they managed to get into the plane to rescue people after a flight attendant opened the rear door after the plane had crashed.
The tragedy, which occurred around 08:50 local time (05:50 GMT), has been blamed on bad weather.
Emergency workers have used ropes to pull the ATR-42 aircraft even closer to the shore, and some of the plane’s body is now above the water.
It is not clear how many people remain trapped inside the plane
Immediately after the crash, the plane was almost completely submerged with only the brown and green tail fin visible.
Tanzania’s Prime Minister Kassim Majaliwa has gone to the scene and said an extensive investigation would be carried out to determine the full causes of the crash.
“We are still identifying the bodies, but it is highly likely that the pilots might have perished,” he said.
The airport has been closed until further notice.
President Samia Suluhu Hassan has expressed her condolences to those affected and called for calm as the rescue operation continues.
The plane was flying from Das es Salaam to Bukoba via Mwanza.
Precision Air is Tanzania’s largest private airline and is partly owned by Kenya Airways. It was founded in 1993 and operates domestic and regional flights.