The Internal Revenue Service is threatening to fine and prosecute Americans who have been making a living in Canada without renouncing their U.S. citizenship.

The IRS says it's going after offshore tax cheats, but some who've been targeted say they have no ties to the U.S. and are simply trying to earn a living as Canadians.

Maury Williams left the U.S. in 1973 and has been a Canadian citizen for 25 years. He's only just learned that American tax authorities have been expecting tax forms from him for decades.

"Why should I be filing back taxes? And, more specifically, why should I be paying a penalty?" Williams said.

His wife Linda, who was born in the U.S., is in the same position. "They have no business looking at our Canadian assets," she said. "We have worked for them and earned them here."

The Williams fear that because their children are also considered U.S. citizens, they will be targeted next.

Including accounting fees, the couple faces a $30,000 expense. And their case isn't unique – but some aren't willing to pay, and are opting to go under the radar instead.

That means they could face civil or criminal prosecution if they try to cross the border.

Staying under the radar is also about to get tougher. New laws set to take effect in 2013 will give the U.S. tax authorities access to personal banking information of everyone they consider to be a U.S. citizen.

The move has prompted a response from Canadian federal Finance Minister Jim Flaherty, who issued a letter to three major U.S. newspapers asking the IRS to back off.

Dual citizens are not tax cheats, or high rollers with offshore accounts, Flaherty told the New York Times, Wall Street Journal and Washington Post.

The minister added that people do not flock to Canada as a tax haven, and the measures are causing unnecessary fear and stress among dual citizens.

With a report from CTV British Columbia's Kent Molgat