The Slovakian, ranked 68th, ousted top seed Halep 6-2 4-6 6-3 in just under two hours on the New Haven hard courts on Tuesday.
Rybarikova broke the Halep serve thrice in succession from 2-1 up in the first set to take the advantage, before the French Open finalist squared the match - leaping to a 3-0 double-break lead before holding onto her lead to send the match to a decider.
Neither player got a look at a break point until the sixth game, but at the first time of asking, Rybarikova converted for a 4-2 lead.
The 25-year-old then denied Halep twice on break point in the ninth game to serve it out on her second match point.
It was a much-needed breakthrough for Rybarikova, who had just one win in seven matches since her run to the semi-finals at 's-Hertogenbosch in June.
Also exiting the WTA International event prematurely was sixth seed Flavia Pennetta, the Italian beaten 6-1 7-6 (7-3) by American Alison Riske.
Riske, the world number 44, hit eight aces in nine service games to ease into the quarter-finals where she will meet Rybarikova.
Czech second seed Petra Kvitova will play countrywoman Barbora Zahlavova Strycova in the last eight, after the pair claimed straight-set victories.
Kvitova overturned a loss to Russia's Ekaterina Makarova in Montreal earlier in the North American swing, downing her fellow left-hander 6-2 6-1.
The two-time Wimbledon champion will face Zahlavova Strycova next, after the latter downed French lucky loser Caroline Garcia 7-5 6-2.