Help me choose a graphics card

I recently started building a retro gaming console, which is nothing more than an old PC with Ubuntu running Retropie.

I actually put this together to test Retropie out before springing for a Raspberry Pi 3, but then I realized that it performed so much better than the Pi 3 that I would be better off not bothering with the Pi at all.

It's an HP 5800 small form factor desktop.
These machines came with an on-board GPU which is no longer supported, but it works great for everything except N64 games.
I borrowed a graphics card from another machine to see if that would help, and it worked perfectly.
Unfortunately, I can't keep this graphics card in the machine for two reasons; it doesn't belong to me, and it doesn't have the right outputs.

I'm hoping that I can get some suggestions for graphics cards that will give me similar performance, but with these features:

HDMI out, including audio.
This is important, because my TV can't accept a separate audio input when using HDMI.
The other video card that I tried had DVI output. I used a DVI to HDMI adapter, but could not get sound because the TV expects sound over HDMI.

Similar performance to the other card I tried.
It was a GEForce 9000 series.
I don't dare go lower than this for fear that it won't have enough grunt.

Cheap!
I want to pay as little as possible. I have $40 left in my budget after buying controllers etc, and I I still have to get a good Bluetooth dongle and a couple of other odds and ends.
I'd like to spend $25 or less if possible. There are loads of graphics cards that look like they might be suitable on eBay at around this price.

Plug and play.
The last card I tried just worked. I'd like this one to be the same. This is supposed to be a fun project, and I really don't want to have to mess around with drivers.
I'm concerned that getting audio over HDMI might need some sort of driver, but it's been a long time since I have done anything hardware related with Ubuntu.

Sorry for the long post, please help!