You can use the Marketcetera exchange simulator at
http://exchange.marketcetera.com/It's setup to work with the Marketcetera Platform, the open-source platform for creating automated trading systems.
The simulator itself is not open-source, the platform is. However, the simulator supports FIX, and you can start with that to test your strategies.
You can also try the NYSE TransactTools OpenFix (
http://www.openfix.net/ ) service. It's not a full simulator, but can be used to do FIX certification of your FIX engine.
QuickFIX/J has a very basic "executor" example as part of the QuickFIX/J distribution:
http://www.quickfixj.org/Full disclosure - I am one of the founders of Marketcetera and a maintainer of the QuickFIX/J project