The answer depends a little bit.
Most SEO and marketing companies will not control your web site.
There are some companies that do require this and hold control of it.
Do you have access to your own analytics or do they provide it for you?
Is it your own domain name? Did they build your web site for you?
If you leave an SEO company you should not experience immediate negative results. Depending on the competitiveness of the niche you may be quickly over taken.
Some companies provide a lead gen service where they deliver traffic to your site or leads to your inbox and thus when you turn off their service you lose all of that.
In nearly all situations you want to retain your domain name and web site. Starting from scratch with a new domain is an expensive proposition.
Message me or send an email through my web site and we can look at your specific situation in a little more detail.