result in even fewer Tory MPs backing a deal in Parliament.
But Jeremy Corbyn insists his support is contingent on ministers accepting the need for a customs union.
Supporters say it would be better for businesses, but opponents feel it stops the UK setting its own trade policy.
Talks between Labour and the government have been taking place for a number of weeks after Mrs May's Brexit deal with the EU was effectively rejected for a third time by MPs.
Downing Street says further talks are being scheduled "in order to bring the process toward a conclusion" - and according to BBC Newsnight's political editor Nicholas Watt, a sense of urgency is growing.
Labour has previously complained that the government appeared unwilling to move on the possibility of a customs union.
But the PM's de facto deputy David Lidington said on Monday that the latest round of talks had been "productive", while Labour described them as "constructive".
All EU members - including the UK at present - are inside the customs union, meaning they do not have to pay taxes, called tariffs, to move goods and services between them.
This keeps cost down and avoids delays, but members have to operate as part of a bloc and cannot do their own trade deals with other countries around the world.
Labour is arguing for the UK to be part of a new form of customs union arrangement where the country could have "a say" in policy despite no longer being in the EU.
Critics say the EU would never agree to that, and in any case, the purpose of Brexit is to break free and remaining in a customs arrangement would go against that.
Leaving the customs union was also a Conservative manifesto commitment at the 2017 general election.
- What are the Brexit sticking points?
- What is Labour's customs union policy?
- Brexit: Significant hurdles for cross-party talks
- Can you have frictionless trade in a customs union?
Mr Hunt told BBC Radio 4's Today programme: "If we were proposing - which I very much hope we don't - to sign up to the customs union, then I think there is a risk you would lose more Conservative MPs than you would gain Labour MPs."
However, he said he "definitely" thought the government could secure a deal that MPs would back.
Other cabinet ministers - including International Trade Secretary Liam Fox - have also recently signalled their unease at a customs union agreement.
Shadow chancellor John McDonnell tweeted that Mr Hunt's comment did "not inspire confidence that if a deal is agreed it would be successfully entrenched and last any longer than the next Tory leadership election".
Nick Boles, an independent MP who quit the Conservatives over the party's Brexit stance, also criticised Mr Hunt's comments as "ignorant and unhelpful", saying on Twitter that it revealed "naked opportunism" over the Conservative leadership.
Mrs May has said she will resign if her deal is passed through Parliament, leading to the prospect of an imminent leadership contest.
The prime minister is keen to get the Withdrawal Agreement Bill - legislation required to implement the withdrawal deal - through Parliament as soon as possible.
Ministers insist it is still their aim for that to happen by 22 May so the country does not have to take part in elections to the European Parliament a day later.
Please upgrade your browser
Your guide to Brexit jargon
Use the list below or select a button