I would like to create a video chat web app, similar to "Chat Roulette", I'd like to use Flash for both receiving and streaming video & audio.
The main problem is that it seems like they always redirect you to purchase a Flash Media Server license and usage, instead of allowing you to host a server yourself.
Is there any good known alternative to that, that I can host by myself and that works well with the flash streaming APIs?
Check out Red5: https://github.com/Red5/red5-server
It's an open source Java-based media server.
There are many other alternatives out there, but since I've only used FMS and Red5, I can't vouch for other solutions.
Some that might also be worth considering:
Please keep support and community activity in mind when choosing a platform, it might come in useful to choose something that has an active community.