Living with one gigabyte of memory
A small VPS is plenty for a static site and a couple of services, as long as you pay attention to what is actually running.
The first thing I do is look at the real numbers rather than guess:
ps -eo rss,comm --sort=-rss | head
free -mThe "available" column is the one that matters. A low "free" number only means the kernel is using spare memory for cache, which is what you want.
Beyond that, a small swap file gives you some warning before the OOM killer picks a victim, and capping journal size stops logs from slowly eating the disk. Most of the rest is just not installing things you do not use.