Moore, making his first start since undergoing Tommy John surgery in April 2014, faced the minimum nine batters over the first three innings before fading quickly.
"I thought there might have been more nervous body-control type issues, but really it felt normal," said Moore, who gave up four runs on six hits in 4 2-3 innings. "But anytime you can put up two runs early, especially against a guy like Cory Kluber, you can't squander that, especially when you're walking guys with two outs . . . I've got to do better than that."
The 26-year-old left-hander gave up two walks and threw three wild pitches.
"A couple pitches got away from him, he yanked a couple changeups," Rays manager Kevin Cash said. "But I think we all came into it knowing he's not going to be fine-tuned in his first outing."
Kluber allowed four runs and seven hits. He settled down after allowing three straight hits and falling behind 2-0 in the first inning. The Rays didn't have another hit until Kevin Kiermaier doubled off the tip of first baseman Carlos Santana's glove leading off the sixth. Kiermaier scored on Grady Sizemore's second RBI single of the game.
Asdrubal Cabrera's infield single - also off Santana's glove - drove in a second run in the sixth, making it 4-4.
Despite setting a team record for strikeouts, Tampa Bay's offense took a small step forward after being shut down offensively over the first three games of the series, outings which saw Carlos Carrasco, Cody Anderson and Danny Salazar take perfect games beyond the fifth inning.
The Rays finished with three hits on Monday, two Tuesday and just one Wednesday, when Carrasco came within one strike of throwing Cleveland's first no-hitter in 34 years before Joey Butler's single broke it up.
"We knew we had our hands full coming into today," Kiermaier said. "The last three days, everyone knows how that went, and to cap it off with Kluber today . . . We did a good job early on, then they came back and he just kept going out there and mowing us down."
A 17-game winner two years ago, Moore was injured during his second start of 2014 and had surgery performed on his pitching elbow two weeks later. He's the ninth starting pitcher used by the Rays this season.
"From the outside looking in, it seems like a tall order going against Cory Kluber, the way their last three (pitchers) have thrown against us," Moore said. "They had it going this series, that's without a doubt."
Jason Kipnis and Francisco Lindor began the Cleveland fourth with singles and Michael Brantley followed with his sacrifice fly to cut Tampa Bay's lead to 2-1. The Indians added three runs in the fifth off Moore.
Indians: RHP Trevor Bauer (6-5) will face Pittsburgh RHP Charlie Morton (6-1) in the opener of a three-game interleague series Friday night. Bauer is 2-0 with an 0.82 ERA in three interleague starts this season. Morton is 4-0 with a 1.98 ERA in four home starts.
Rays: RHP Chris Archer (9-5) and New York Yankees RHP Masahiro Tanaka (4-3) are set to pitch in the first of a three-game set Friday night. Archer gave up a career-high three homers against Boston in his previous start Sunday.