A high-rise building under construction collapsed in Nigeria's commercial capital Lagos on Monday, killing at least three people with dozens more feared trapped inside the rubble.

A yellow excavator pushed away concrete slabs in the search for people in the  wreckage of the 21-floor building in Lagos's Ikoyi district, AFP correspondents at the scene said.

Rescue officials said "many" workers were caught inside the building when it crumbled, though they could not confirm the number of people trapped or dead.

Dozens of angry residents gathered at the site soon after the collapse, many crying and voicing frustration over the slow pace of the rescue efforts.

Lagos State police commissioner Hakeem Olusegun Odumosu said three bodies had been recovered so far and three people rescued.

"Three corpses have been recovered and the operation still goes on," he told reporters. 

He said it was too early to say what had caused the building to collapse.

"Many workers are trapped under the rubble," said Femi Oke-Osanyintolu, general manager of Lagos State emergency management agency.

Soldiers kept back a crowd of onlookers watching the rescue operation at the site, where concrete floors of the building were sandwiched together.

Four construction workers at the site told AFP dozens of their colleagues were inside when the building crumbled. 

"Like 40 people were inside, I see 10 bodies because I climbed up," said Peter Ajagbe, 26, one local worker on the site.

"One of my partners is dead."

Taiwo Sule, 21, another worker, said he had seen five bodies on top of the collapsed building, where he had tried to help recover them. 

An AFP reporter at the scene saw at least one person being pulled unconscious from the rubble.

"I feel so bad because the people that are inside, they have family," said construction worker Latif Shittu. 

The Ikoyi area is one of the wealthier residential and business districts in Lagos, Nigeria's densely populated major commercial city.

Building collapses are common in Lagos and other parts of Africa's most populous country because of the use of sub-standard materials, negligence and the flouting of construction regulations.

In one of the worst building disasters, more than 100 people, mostly South Africans, died when a church guesthouse crumbled in Lagos in 2014.