The top official in Swat, Kushal Khan, told the BBC a curfew had been lifted to allow residents of four areas close to Mingora to leave.
He said the Government was setting up a camp elsewhere in the district to house them.
The Government later rescinded its earlier order to encourage them to leave but the residents continued to flee.
The army says militants have attacked police checkpoints in four different locations around Mingora, the district headquarters of Swat.
A Pakistani military spokesman, Major-General Athar Abbas, says it is clear the militants are ignoring the peace deal, which has been in place since February.
"The militants have violated all the norms of the peace agreement," he said.
"The military has conducted the operation on the orders of the Government to clear the militants from the valley then also take control of neighbouring districts, push them out, eliminate them and restore the rule of the Government."
Tension in Swat has steadily worsened over the past week after the army accused the Taliban of breaking the peace deal and of trying to take over neighbouring areas.
Provincial Information Minister Mian Ifthikar Hussain said at a press conference in Peshawar that the number of people displaced from Swat could rise to half a million. He said a large number of people were leaving the valley as the situation worsened with the Taliban refusing to disarm and rejecting an agreement they had earlier signed for the implementation of Sharia courts.
Militants have swarmed into Mingora since Sunday after rejecting a government move to operationalise a February agreement with the militants to set up Sharia courts in the region, demanding instead that the military should first halt operations in neighbouring Buner district. A military operation in Swat seemed imminent with several clashes already between militants and security forces in and around Mingora.
On Tuesday, Taliban militants continued a day-old siege of a power station on the outskirts of Mingora with 46 paramilitaries trapped inside. Fighting between security forces and militants was also reported from another part of Mingora.
The militants have occupied a few of the main buildings in the town, including a well-known school, a mosque and hotels. A resident told The Hindu by phone that they were openly moving about in the main roads of the city, brandishing weapons. After the Taliban blew up a power station, there has been no electricity either.
“There’s an atmosphere of fear and panic here. People don’t know what’s going to happen next,” said the resident, who did not want to be named. Adding to the problems was the heavy rain on Tuesday.
The resident, a lawyer, said hundreds of people started to flee Mingora after an announcement by a top district official asking people to leave for their own safety. They were heading towards Peshawar, Mardan, Rawalpindi and Islamabad, he said.
“People took any means possible. They went in their private vehicles, in buses, trucks,” he said. In many cases, he said, one male member of the family remained to look after property.
An indefinite curfew declared on Tuesday morning was relaxed to allow people to leave. Curfew was imposed again at 5 p.m. for an indefinite period. In Peshawar, meanwhile, an explosives-laden car rammed a security vehicle killing four civilians and one paramilitary soldier. Twenty-one people were injured, among them several children waiting to be picked up by their school van.
On Wednesday, President Asif Ali Zardari is to meet his U.S. and Afghan counterparts Barack Obama and Hamid Karzai for discussions on the strategy unveiled by the Obama administration to defeat militancy and terror in the region.
Despite reports that under U.S. persuasion opposition leader Nawaz Sharif would join the Pakistan People’s Party-led government to strengthen its hands in the fight against militancy, this does not seem likely at least for the moment.
Following a meeting between Prime Minister Yousuf Raza Gilani and Punjab Chief Minister Shahbaz Sharif in the capital on Tuesday, the PML(N) leader said the power sharing arrangement between the two parties in the Punjab province would continue but expressed the party’s reluctance to join the government at the centre.
“We have some unresolved issues between us, such as the amendment of the Constitution [to remove changes made by the former President, Pervez Musharraf]. But we are not running away anywhere. We are right here and we will offer all support to the government,” said Mr. Nawaz Sharif at a press conference.
He also reiterated his call for an all-party “national conference” to discuss ways of resolving the crisis in the North-West Frontier Province, saying the problem was too huge for any one political party to try and resolve on its own.