Top bankers warned on Tuesday they could start moving staff abroad as early as next year if there is no clarity on whether Britain will retain access to the European single market when it leaves the EU.

Senior executives from European divisions of some of the world’s biggest financial institutions told a conference in London they felt the government’s tougher rhetoric on immigration risked harming the economy.

James Bardrick, the UK head of U.S. bank Citi, said the main dilemma facing the finance industry was how urgently it needed to act on contingency plans aimed at protecting their businesses, following Britain’s vote in June to leave the European Union.

“How do we and when do we start making decisions … knowing the plan is ready to go … it could be in the first quarter of 2017,” he told a conference in London.

British Prime Minister Theresa May has said she would trigger the two-year process to leave the EU by the end of March and last week appeared to prioritize capping immigration over retaining access to the single market.

The future of London as Europe’s financial center is expected to be a major negotiating point in May’s talks with EU partners, with banks keen to retain the “passporting” rights which allow them to sell financial services across the bloc.

Getting the best deal for the City of London will be an “absolute priority” in Brexit trade talks with the EU, financial services minister Simon Kirby told the conference.

Rob Rooney, CEO of Morgan Stanley international joined the chorus of bankers concerned about passporting, and said his bank would also have to move parts of their operations from London if Britain were shut out of the single market.

“It really isn’t terribly complicated. If we are outside the EU and we don’t have what would be a stable and long-term commitment to access the single market then a lot of the things we do today in London, we’d have to do inside the EU 27,” he said.

Separately, Jacob Rees-Mogg, a Conservative lawmaker and member of the Treasury Select Committee who campaigned for Brexit, told Reuters that banks needed to take responsibility for their own affairs in line with their contingency plans.

“If the City has to do anything, it needs to do it now. I don’t think it needs to do much but it needs to get on and do it,” Rees-Mogg said, when asked about the threat of a major bank exodus from Britain.

“Bank lobbyists are going down the wrong route on Brexit. They just seem to be whingeing. I don’t see what they’re moaning about.”

Culture Hit

Tim Roberts, managing director of consulting firm Promontory Financial Group, said London began to rival New York as a financial center in the 1980s because it welcomed people regardless of their nationality or gender.

“If we lose that, and we allow not Brexit itself, but some of the sentiments expressed nationally that led up to the referendum to erode that culture then I think it damages London as a financial center,” Roberts said.

The government backtracked on a recent proposal to make companies list their foreign workers after an outcry from business groups, who said any initiative to “name and shame” employers would be divisive and discriminatory.

The chairman of the Lloyd’s of London insurance market John Nelson criticized the tone of language used by government ministers, and said Britain risked turning its back on the global economy for the first time since World War Two.

“We have to be very clear that the rest of the world is looking very, very closely right now,” Nelson said.

“We have to be careful that we keep the UK open… and making sure we are able to attract the right talent globally. If we don’t do that we will not remain the financial sector that we are today.”