At the height of the operation, five aircraft from four different nations with Royal Navy warships were involved in the search for the mystery vessel, according to a report by Aviation Week.
However the report said that it was unclear whether they succeeded in locating the submarine, or whether any foreign submarine had entered UK territorial waters.
The Ministry of Defence (MoD) confirmed that it had received assistance from Nato allies but would not say whether they had been searching for a submarine.
The hunt was said have been launched after a periscope was spotted in waters where Royal Navy submarines normally surface as they head into or out of their base at Faslane late last month.
At one point it involved two US Navy P-3 Orion aircraft, a CP-140 Aurora from the Royal Canadian Air Force and a French navy Dassault Atlantique 2, operating out of RAF Lossiemouth.
An RAF Sentinel radar-reconnaissance aircraft was also said to have taken part in the operation which appeared finally to have drawn to a close last week.
Since the Government scrapped its Nimrods in the 2010 Strategic Defence and Security Review, the UK has lacked a specialist maritime patrol aircraft.
A MoD spokesman said: “Nato partners have provided assistance for the operation of maritime patrol aircraft from RAF Lossiemouth for a limited period with the Royal Navy. We do not discuss the detail of maritime operations.”
The incident comes just weeks after Sweden mounted a search for a suspected Russian submarine thought to be operating in its territorial waters.
Although the hunt was unsuccessful, defence officials said there was no doubt that their waters had been violated by a foreign power.
Last month the Royal Navy tracked four Russian warships passing through the English Channel while there has been a recent upsurge incidents of Russian long-range bombers approaching UK airspace.