The British surveillance program was first revealed by Edward Snowden, the former USA defense contractor who leaked similar information about the National Security Agency in 2013.
The mass spying scheme did "not meet the "quality of law" requirement" and was "incapable of keeping the "interference" to what is "necessary in a democratic society", the ECHR said.
It also ruled that the regime covering how the spy agency obtained data from internet and phone companies was "not in accordance with the law".
Judges sitting in Strasbourg, France, said GCHQ's bulk interception of online communications was untargeted and criticized spy bosses for failing to provide sufficient safeguards when handling people's personal data.
Brought before the European Court of Human Rights by Big Brother Watch, English PEN, and the Open Rights Group following the revelations by whistleblower Edward Snowden that the United Kingdom and other countries had been conducting secret population-scale communications interception against their citizens, the cases were joined together for a single hearing.
In 2013, Edward Snowden revealed that GCHQ - the UK's eavesdropping agency - had been secretly collecting communications sent over the internet on an industrial scale.
Three applications were joined together, from Big Brother Watch, the Bureau of Investigative Journalism, and 10 human rights charities, and were lodged after Mr Snowden's revelations.
When it came to requests for data from communications service providers under Chapter II, the Court noted that the relevant safeguards only applied when the goal of such a request was to uncover the identity of a journalist's source.
Britain broke human rights law when the GCHQ intelligence agency carried out the mass snooping operation that was exposed by Edward Snowden, a European court has ruled.
The ECtHR found, by five votes to two, that the bulk interception regime violated Article 8 of the European Convention on Human Rights, which covers citizens' rights to have their private life and communications respected.
Civil liberties groups are set to continuing fighting a number of cases related to surveillance and while rulings have gone against the government, nothing has changed; the Investigatory Powers Act is still the same as it was in 2016 and the government doesn't look to be in any rush to alter it, despite these losses. Today, the Court has ruled the UK's surveillance illegal, despite arguments that the Regulation of Investigatory Powers Act 2000 (RIPA) and its replacement the Investigatory Powers Act 2016 (IPA) made it legal. "It can and must give us an effective, targeted system that protects our safety, data security and fundamental rights".
The court's decision does not prohibit governments from sharing information with other countries, because there was no evidence that part had been abused.
Although the case considered procedures governing bulk cable-tapping that are no longer in force - since replaced by the Investigatory Powers Act - campaigners have hailed it as a further nail in the coffin of state surveillance.