Were we moving too fast?

Speak to him. Don't make things awkward. Just tell him you're not comfortable with having sex. If he is a respectable guy he will be okay with that. Don't ignore him because you feel uncomfortable. COMMUNICATION