Hayes scored at 3:14 of overtime to give the Rangers a 2-1 victory Wednesday night and a 3-1 lead in the best-of-seven Eastern Conference first-round series.
"It will be a memorable one, for sure," Hayes said after his first playoff goal.
Martin St. Louis started a goalmouth scramble when he threw the puck in front on a wraparound. The puck pinballed to Carl Hagelin and went across the crease for Hayes to flip it past fallen goalie Marc-Andre Fleury.
"I thought two of our guys worked hard and got the puck to the net and it just kind of ended up on my stick," Hayes said.
And he buried it.
But it's nothing new for the rookie, who helped the NHL's best regular-season team to its first Presidents' Trophy since 1994 with 17 goals and 45 points, including a team-best 29 points since the All-Star break.
"This was his best game of the series," Rangers coach Alain Vigneault said. "We need him to play the way he's played for us throughout the year. Obviously, this is playoff hockey on a bigger stage, but he's learning how to deal with it."
Game 5 is Friday night in New York.
New York's Derick Brassard tied it at 1 with 2:45 left in the second period, scoring his third goal of the series. Patric Hornqvist opened the scoring at 2:22 of the first period for Pittsburgh.
Fluery and New York's Henrik Lundqvist each made 22 saves.
The Penguins, who have two or fewer goals in 13 of their last 19 games, are a loss away from elimination after dropping consecutive home playoff games to the NHL's best regular-season road team. The Rangers have won 12 of their last 13 on the road, allowing 15 goals.
Nine of the last 11 playoff games between the two teams have been decided by two or fewer goals, including four one-goal games in this series.
"We need to make sure that we're desperate," Penguins captain Sidney Crosby said. "Really the only thing you can do is go out there, be desperate and get it back here."
Hornqvist nearly ended it in the final minute of regulation, but Lundqvist denied him with a sliding save and Dan Girardi swept the rebound away from a gaping net.
"If you look at the way the whole series has gone, it's just a fine line," Penguins coach Mike Johnston said.
The Penguins got the start they wanted when Hornqvist redirected Ben Lovejoy's point shot through Lundqvist's legs. They dominated the first 10 minutes, peppering Lundqvist with the first six shots.
It was a stark contrast from the first three games of the series when New York outscored Pittsburgh 4-0 in the opening period and limited the sluggish Penguins to 13 shots during that span.
It took the Rangers 11:35 to register their first shot on goal Wednesday, one game after Pittsburgh needed 15:10 for its first shot of the game.
"I liked the way we started, but we could've had a couple more," Johnston said.
The Penguins continued to press in the second period, but couldn't pad the lead. New York, which scored the opening goal in the first three games of the series and held a 2-0 lead in two of its wins, tied it when Brassard poked at the puck until it went across the goal line. Rick Nash started the sequence with a bad-angle shot from the corner that caromed off Fleury and fell to the ice.
"There's not a lot of room on the ice and both teams are battling extremely hard," Vigneault said. "We were able to tie it up in the second and get it done in overtime."
NOTES: Evgeni Malkin became the fifth player in Pittsburgh history to appear in 100 playoff games. Sidney Crosby played in his 99th and Fleury his 97th. Malkin has gone nine straight games without a point and 14 in a row without a goal. ... The Penguins lost four straight home playoff games to one team for the first time in team history. ... D Derrick Pouliot (upper body) sat for the Penguins and D Kevin Klein (broken arm) for the Rangers, though both are close to playing. ... F Beau Bennett, F Craig Adams, D Scott Harrington and D Christian Ehrhoff were scratched for the Penguins, while the Rangers sat D Chris Summers and C James Sheppard. ... The Rangers entered the game 1-12 in games played after taking a lead in a playoff series since 2012 and had lost four of five with a chance to take a 3-1 lead.