Jeff Pope knows more than most about TV drama based on real-life stories, but he can't quite believe the success of Cilla. Lauded by critics – not least for Sheridan Smith's "stunning" performance as the young Cilla Black – ITV's three-part biopic was also a ratings triumph.

Averaging 8.3 million viewers and a 31% audience share, it is the most-watched new drama this year on any channel. That is almost twice the audience for Not Like That, Like This - ITV's much-praised film about Tommy Cooper, broadcast in April.

Cilla also beat ITV's other hit fact-based drama, The Widower, about a convicted murderer, which averaged 7.5 million viewers.

"After decades of making dramas about serial killers, murderers and heavyweight politicians, the life of a girl pop star seemed something of a departure," Pope admitted before Cilla was shown. As writer, executive producer and Head of Factual Drama for ITV Studios, his successes include See No Evil: The Moors Murderers, the Fred West biopic Appropriate Adult, Lucan, Pierrepoint, Mrs Biggs, Mo and Philomena.

"I had always yearned to do a story like Cilla," he says. "The chance to capture performance as well as drama was an ambition for me – and with Cilla, there were some iconic songs to choose from."

Pope also had the perfect Cilla, after working closely with Sheridan Smith on Mrs Biggs, which won her a Bafta.

"Sheridan worked for months to get her voice in shape for the rigours of a two-month shoot and to get that 'Cilla' sound. I think the result is astonishing," says Pope.

But that wouldn't have been enough without a compelling story, he insists: "I was determined that it shouldn't be a hagiography, it had to have teeth – and one of the most interesting things was to discover how ruthless Cilla had been in putting her career before her boyfriend, Bobby Willis.

She told me: 'There were times when I was a real cow.' Another surprise was that her manager, Brian Epstein, wanted her to go into television and she didn't.

"This fascinated me, the idea of a pre-Blind Date/Surprise Surprise Cilla running a mile from the career that (for those under pensionable age) now defines her," he adds.

Pope is a former journalist, famed for the rigour with which he researches his subjects. Before Cilla began shooting, he spent many hours with the star, talking about her story.

"We got to the point where she gradually trusted me enough to reveal some intimate and never previously discussed details," Pope recalls. "And she was brave enough to give her blessing to a picture that doesn't flinch from some of the darker episodes in her life."

But for a long time Black couldn't bring herself to watch the films.

"We gave her a rough-cut version to watch six weeks before broadcast and she was very nervous about it," says Pope.

"Fortunately, her son Robert, who's also her manager, knew there'd be press interest, and he took the DVD round and they watched it together. I've spoken to her since and I know she was pleased."

Pope says the huge success of Cilla means broadcasters are inevitably looking for more dramas based on the lives of celebrities.

The irony is that – with an entertainer as its subject, rather than a murderer – it was more in the BBC drama mould than that of ITV.

BBC Four blazed a biopic trail with a stream of films about popular entertainers and successful women (sometimes both). They include Kenneth Williams, Tony Hancock, Frankie Howerd, John Lennon, Hattie Jacques, Fanny Craddock, Enid Blyton, Gracie Fields, Margot Fonteyn, Isabella Beeton, Hughie Green, Harry Corbett and Wilfrid Brambell.

The road ran out in 2013 with the award-winning Burton and Taylor, who were played by Dominic West and Helena Bonham Carter.

The BBC axed BBC Four's dramas to cut costs, even though most were made on a shoestring, understood to be around £500,000 an hour.

"The BBC Four biopic has been cut off in its prime," wrote Ben Lawrence in The Daily Telegraph. Sam Wollaston lamented the loss of "these brilliant BBC Four biopics" in The Guardian. ITV's Pope says that, as a viewer, he was "bitterly disappointed".

Others thought the genre had passed its sell-by date. "Bored with biopics" was the headline on a 2010 feature in The Independent, in which Gerard Gilbert argued that life stories were taking huge chunks of drama budgets – and were generally safe and unchallenging.

By then, however, the BBC's drama department was moving on. In 2011, Eric and Ernie was broadcast on BBC Two, starring Victoria Wood as Eric Morecambe's mother, Sadie, with a budget twice that of the BBC Four films.

Ben Stephenson, the BBC's Controller of Drama Commissioning, says "Eric and Ernie took things to a different level, winning 6.5 million viewers, BBC Two's biggest-ever drama audience.

"On BBC Four, those films might get 1 million. Biopics are now in British telly's DNA – but the BBC and ITV have become slightly more selective, and they now feel more 'flagship'."

The key is to do the research, be respectful and keep it authentic

Eric and Ernie was written by Peter Bowker, who also struck biopic gold recently with Marvellous on BBC Two, the true and uplifting story of Neil Baldwin, the kit man at Stoke City, who overcame learning difficulties. It won rave reviews and was watched by 2.2 million viewers.

For both films, it was important to gain the families' support to help inform the story, even though they would have no editorial control.

That can be difficult when so many of the entertainers are shown to have skeletons in their cupboards - alcoholism, lies about their sexuality, complicated love lives and cruelty to children and partners.

Bowker says it was easier with Eric and Ernie because there weren't any great character flaws. "The drama's tension lay in things that everyone can recognise – disappointment, bereavement, an unexpected pregnancy – and the disaster of their first TV show.

"The most sensitive area for me was that Ernie's wife, Doreen, felt his role in the partnership had always been underplayed – the eternal fate of the straight man.

"And because Gary Morecambe has done such a great job in preserving his father's memory, I wanted to redress the balance a bit."

ITV's Tommy Cooper film was tougher, showing him torn between his love for his wife, Gwen, and his assistant, Mary.

The network's Director of Drama, Steve November, says criticism of it came more from Cooper's fans than his family. "There were fantastic performances from David Threlfall and Amanda Redman, and his fans didn't want to hear it, even though it was truthful," he says.

But can writers and producers play with the truth? "It's very important to capture the spirit of events, if not the exact letter," says Bowker. "You can't usually know exactly what was said."

"Above all, you've got to be fair – fair to the spirit of what happened," says Pope. "But you have to condense events and, occasionally, people. For example, Fred and Rose West were interviewed by more than 30 police officers over many months. In Appropriate Adult, I had to reduce that to three or four or the audience would never have followed it."

What if people don't think it's fair? "We'd normally have that conversation at the script stage and then again when editing, before we transmit," he replies. "You don't give away editorial control, but you do want to be accurate and sensitive."

Not all writers and producers are as rigorous as Pope and Bowker, but Ofcom says it generally receives few complaints that biopics are misleading. Stephenson says: "I get more complaints about dead animals than biopics!" "The key is to do the research, be respectful and keep it authentic," says Katherine Lannon, who produced the dramas about Isabella Beeton, John Lennon, Shirley Bassey and Shappi Khorsandi, and script-edited Hawking.

Now an executive producer at Ecosse Films, she says many biopics become "passion projects": "I have been amazed by the dedication of casts and crews to replicate a person, a place and a period – even on the more challenging budgets."

So who's next for the biopic treatment? Pope is working on a comedy drama series for BBC Two based on the autobiography of Danny Baker, with whom he worked on LWT's The Six O'Clock Show in the 1980s.

The BBC is focusing on less well-known people. It has just filmed The C Word, adapted from Lisa Lynch's inspiring blog about her experience of cancer. It stars – who else? – Sheridan Smith.