Lester fired the League Two side in front with his 15th goal of the season, but could not beat Coyne from the spot with a minute remaining.
Chris Greenacre had slotted home a first-half spot-kick to bring Rovers back into the game and Antony Kay won the first round tie in style with a stunning volley seven minutes from time.
Defeat was hard on Chesterfield who had more than matched the visitors and had dominated most of the second half.
It would have been a different story before the break when the League One side were on top until Lester pounced in the 29th minute to drive a left-foot shot past Coyne from 15 yards after Steve Fletcher had flicked the ball on.
But a reckless challenge by Aaron Downes on Chris Shuker eight minutes later allowed Greenacre to equalise, but Chesterfield moved up a gear in the second half and Tranmere had to defend desperately at times.
Coyne did well to keep out a close-range header from Fletcher and both Peter Leven and Gregor Robertson saw long-range shots go narrowly wide.
Lester was also close with a curling shot from the edge of the area, but Tranmere stunned the home side in the 83rd minute when Kay met Paul McLaren's corner with a left-foot volley from 15 yards that gave Barry Roche no chance.
But there was more drama to come when referee Steven Cooke awarded a penalty for a push by substitute Calvin Zola on Downes, but Coyne dived low to his right to beat out Lester's firmly struck spot-kick.
There was still a late anxiety for the visitors when substitute Craig Curran was sent off for a second yellow card, but they held on to go into the hat for the second round of football.