All software - whether an application or plugin - utilizes virtual memory. Generally speaking, software does not know about or care about the amount of RAM on the computer. Increasing or decreasing RAM will not affect the features or capabilites of an application.
Virtual memory is an imaginary block of memory that is 4 GB on 32-bit computers, and is nearly unlimited in size in 64-bit computers. Every application has its own block of virtual memory.
The operating system (OSX, Windows, Linux, etc) takes care of all the work to convert that virtual memory into the physical RAM on the computer. For example, an application can store and refer to 4 GB of data, even though the computer has 2 GB of RAM. Not only that, you can run six different applications at the same time, each of which has its own 4 GB of data it is utilizing, and they will all execute okay on just 2 GB of physical RAM.
The magic to make that happen is a "page file", which is a disk file that the operating system uses to move application data in and out of the physical RAM.
When an application reports "out of memory", it does not mean that the RAM is too small, instead it means that "I am hitting the 4 GB limit of my virtual memory". Adding RAM will not fix that. Closing other applications will not fix that. What will fix it? Here are a few solutions: (1) shift to a 64-bit computer and use a 64-bit application; (2) On Windows: use the "3 GB" option in your boot-up file; (3) use a "bridge" plugin such as J-Bridge or VE Pro (which effectively gives you two applications, with 4 GB each); (4) Try to reduce data usage by the application, for example, close unused plugins; for Trilian: make sure you have Streaming turned on on the System window; turn on "Lite" choice in the patch browser; reduce the Pre-Load knob on the System window; finally: Trilian and Omnisphere include a "Lite" option that will only load the minimal samples necessary to support the notes that are in the song your host is playing. Refer to the Reference Guide for details.
If virtual memory is so important, why bother installing more RAM? Because undersized RAM means that the operating system will have to frequently move lots of data in and out of the page file: lots of disk IO means slower execution speed and possible audio glitches.
Conversely, adding RAM will not eliminate the "out of memory" errors from hosts, but it will make things run faster and may prevent autio glitches.
If you have a 64-bit system, and a 64-bit host, and a 64-bit plugin, you will never see the host issue an "out of memory" error, because virtual memory is nearly unlimited. However, you will see performance problems if you load up lots of plugins and lots of data, and your RAM is under-sized.