Ok lets go.
Firstly if I start to say stuff that you know well about and you think 'jesus this guy, I'm not an idiot' then my apologises upfront
Working on testbench is brilliant ! That at least shows your behavioural model is ok. When it comes to the gatelevel in the FPGA we have to remeber timing is going to come into play.
Few things to get us warmed up.
Make sure that ALL your used outputs and inputs in the design go to a FPGA pin. At synthesis if an logic output dosent go to a pin then the synthesis tool will consider that logic redundant and optimise it away ! Like wise an unconnected Input will get connected to either '1' or '0' internally by the synthesis.
Check the synthesis report for logic being optimised away. Also check the synthesis tool to see that its infering the logic that you expected. You said your design was statemachine based make sure that the synthesis tool is picking up that you have a state machine, it should tell you in the report that is sees a statemachine with x ammount of states. Confirm that you do indeed have that many states in your design. Also check to make sure the tool isnt flagging that you have unreachable states.
Clocks & Reset.
Maybe some basics here, again sorry if I'm preaching. In a testbench its easy to generate a clock. Where in you design is the clock ? Is it internally generated or fed in from outside ? On clocking its better to always only stick to using only one clock edge (pos or neg, dosent matter) In testbench you will be fine cause you have made a nice 50% duty cycle clock but in real life a xtal oscillator clock maybe as bad as 80/20 duty cycle. This will give you more headaches than its worth if you are trying clock stuff on different clock edges.
TD
R Pick a clock edge and stick to it
I hope to god you are using a GLOBAL reset to get all your logic into a nice state ? Also for the love of God please use a synchronous reset in your design.
In test bench resets are easy peasy....How are you generating a reset for your FPGA ? A good way is to use a external push button switch connected to a FPGA IO. This signal will not be synchronous to your system clock as you could press it at any time, as this signal comes into your FPGA, actually let me start that again.
Any signal that comes into your FPGA that is NOT based off the system clock is asynchronous and needs to be made synchronus in order to avoid metastability.
Easiset thing to do is to pass the Async signal through 2 D-type flipflops.
Simulation Models.
I assume its behavioural level that you did in modelsim. In the quartus tools it should generate for you other simulation models
after synthesis
after place/fit/route in the FPGA
These would be worth simulating to see if your design has any timing issues. The end result simulation model will have the timing due to the internal routing within the FPGA and any logic delays.
This should be as close to what you see on the bench in real life.
Timing Constraints.
Do you have any timing constraints set ? At the most basic level you need to at least let the tools know your system clock speed. The tools needs this info when doing place and route in order to make sure all the logic meets setup and hold timing internally. Check for a timing report and see if you have any setup/hold violations. the timing report should also give a maximum speed your design can be clocked at. if the timing report says max 25Mhz and you know that you are using a 80Mhz clock then you know you are gonna have issues. Conversly if he reports max 80Mhz and your actual clock is only 25Mhz then you can be pretty confident that you will be ok.
Timing violations and issues can be tricky to resolve but usually pipelineing is alls thats needed. Pipelineing is adding flip-flop stages to a signal in order to break up a long timing path, it addes a clock delay to your signal but you can work with that.
Without seeing any code, reports, block diagrams etc its hard to know what to point at art.
But to start with....Clocks, resets, synthesis reports !
A handy debug idea is to bring out your current state signals (say currentstate[4:0] then bring those 5 signals out to IOs and scope them.... Apply reset to your logic and see if the current state goes through the states you expect ! if it stays static then is your statemachine getting the right inputs in order to make it progress to the next state ? !
On statemachines always have a state that you goto after a system reset ! It makes debugging Statemachines much easier !
I've thrown a bit out here and I'm sorry if its too much or even if its all old news to you.
let me know how you are getting on with the above and maybe it rings some bells as to where to look further. Feel free to ask any questions or if you want me to go into more detail on anything.