Level 1

95 points

EXS in L8 versus L7: no more 3.5 GB RAM limit (with screen grabs)

okay

i’ve done some testing of EXS in Logic 7.1.1 and Logic 8. while all of this testing originated in the “is logic 64 bit” thread, i’ve started a new thread to post these results since, honestly, i really have no idea where to post in that old thread anymore. suffice it to say that Orren and Rohan are absolutely correct -- if we use Virtual Memory in EXS in Logic 8, we are no longer limited by Logic's 3.5 GB RAM limit for EXS and we can now use all of the RAM in our computers with EXS and Logic.

a couple of notes:

1. all songs were created and/or opened on a fresh system directly after a reboot. i never reopened a song or switched songs without a reboot in between. and all testing was done on my dual G5 with 8 GB of RAM, OS 10.4.10, Logic 7.1.1 or Logic 8.

2. i never had any programs other than Logic, Activity Monitor, and Grab open.

3. all testing was done with the EXS Virtual Memory option Active and set to “disk drive speed: slow” and “disk recording activity: extensive”. one of the confusing things in reporting this testing has always been that both EXS and Activity Monitor have a thing called “Virtual Memory” and they are not the same thing. i use Activity Monitor VM as my “how much RAM is this program using” reference because it is the one consistent figure i can find: when Logic hits 3.55 GB of Virtual Memory space in Activity Monitor, it crashes. but Activity Monitor’s Virtual Memory figure is a completely separate thing from the EXS Virtual Memory option. so from here on, when i refer to VM, it is the figure as reported by Activity Monitor. please know that the EXS VM option was active for all testing reported here.

so i started by creating a song in Logic 7 and i added EXS instruments (and ONLY EXS instruments) to the song until i was about to hit the 3.5 GB Virtual Memory limit. as you can see in this screengrab...

...once i had loaded 18 EXS Vienna Symphonic Library instruments, Logic’s Memory figures in Activity Monitor were:

Logic Pro 7.1.1

Real Memory: 3.08 GB
Shared Memory: 47.4 MB
Virtual Memory: 3.42 GB

the system was using 4.15 GB of RAM, leaving 3.85 GB of RAM free. when i tried to load a 19th EXS VSL instrument, the program crashed.

so i then rebooted and opened this same song in Logic 8. as you can see in the screengrab, the memory figures as reported for Logic 8 are very different:

Logic Pro 8

Real Memory: 1.94 GB
Shared Memory: 562 MB
Virtual Memory 2.45 GB

Logic’s Real and Virtual memory figures are much smaller, but the Shared memory figure is much larger. also, the system is using almost the exact same amount of RAM: 4.13 GB (leaving 3.87 Free).

i then proceeded to add EXS instruments (and ONLY EXS instruments) to this song (it has a new name because i didn’t want to save over the original song) until the System RAM was almost all used. as you can see in the following screengrab, the song now has 32 EXS VSL instruments loaded and the system is using 7.68 GB of RAM.

what’s interesting is that despite all of the added EXS instruments, Logic’s memory figures have seen little change. they are:

Real Memory: 2.12 GB
Shared Memory: 571 MB
Virtual Memory: 2.62 GB

and while Activity Monitor isn’t showing a process that is responsible for storing the EXS samples, all 14 of the new EXS instruments are playable (as are the original 18).

so clearly what Rohan and Orren reported about EXS in Logic 8 is true -- we can now load/stream more instruments in EXS in Logic 8.

to test further, i removed a couple of instruments from this song so that i wouldn’t be running quite to the edge of my system RAM, saved it, rebooted and reopened the song. as you can see in this screengrab...

there are now 28 EXS instruments open and Logic’s memory figures are reported as:

Real Memory: 2.07 GB
Shared Memory: 562 MB
Virtual Memory: 2.52 GB

The system is using 6.75 GB of RAM, leaving 1.25 GB free.

so then i added 7 Altiverb plug ins to 7 channels in Logic 8 (different settings for each altiverb instance so they were each using their own RAM).

as you can see, not only has the System used more RAM, but Logic’s memory figures have increased as well because Altiverb is loaded into Logic’s RAM. Logic’s RAM figures now are:

Real Memory: 2.63 GB
Shared Memory: 582 MB
Virtual Memory: 3.15 GB

so i then proceeded to add two more EXS instruments to this song and you can see that the System is now using a lot more RAM (around 750 MB more RAM) but Logic’s memory figures have remained almost the same:

Real Memory: 2.65 GB
Shared Memory: 587 MB
Virtual Memory: 3.16 GB

and, of course, all 30 EXS instruments are playable. how well they will perform in real world use remains to be seen. but they all work.

finally, as near as i can tell, Logic 8 doesn’t let EXS have its own memory space until Logic hits around 1.75 GB of Virtual Memory (as reported in Activity Monitor). before that point, it appears to me that Logic keeps EXS “inside” the Logic memory space. the “real” and “virtual” memory figures for Logic continue to rise as EXS instruments are added and the “shared” memory figure for Logic stays at around 40 MB.

but after 1.75 GB of VM, the “shared” figure starts to rise with each instrument until Logic hits around 2.4 GB of VM, at which point, all EXS appear to be loaded into non-Logic RAM because the “used” System RAM increased but all Logic figures remain almost the same. (i’m sorry if describing this is confusing -- i’m doing the best i can)

anyway, i apologize for the long post, but i wanted to clarify as best i could what the state of EXS is on Logic 8 for anyone still curious.

while i certainly make no claim to know how or why this is happening, i can say that on my system, what Rohan and Orren have reported is correct: EXS in Logic 8 is no longer limited to Logic’s 3.5 GB RAM limit so long as you have the EXS Virtual Memory option active (if it isn’t active, the old RAM limit remains).

thanks to all who helped get to the bottom of this. i hope this post is helpful in demonstrating what EXS can now do in Logic 8. if not, well, i tried. 😉

cheers

Posted on Sep 18, 2007 12:49 PM

103 replies

Justin C

Level 6

8,700 points

Sep 24, 2007 1:50 PM in response to Mike Connelly

Regardless of Leopard's delay, I find it a bit odd how low adoption of this was in Tiger. There are some users really demanding this feature. Even NI's Kontakt 3 will not support this (out of the gates).

J

xs4is

Level 4

2,800 points

Sep 24, 2007 1:55 PM in response to Mike Connelly

Hi all.

Let me interject here and point out a couple of poorly worded points.

First, the EXS doesn't access more then the 32 bit RAM allocation, it's multiple EXSs that each can access the 32 bit RAM allocation on their own. In other words each EXS can access 3.5 gigs of RAM or so. (BTW, I have it on good authority that on Leopard, the standard Logic 8 does go past the 32 bit allocation...don't ask Mike, that's as far as I go down that road).

Second, the word "hackery" or any other similar word implies that it's broken or they're cheating somehow. This is just an incredibly well designed, well thought out way to access more RAM than a 32 bit Logic can. Calling the process "hackery" is kind of insulting.

X

Sep 24, 2007 2:21 PM in response to xs4is

i agree. additionally, there are performance hits to using 64 bit. i think i would rather logic run as efficiently as possible and use 64 bit processes where absolutely necessary and give us the maximum performance possible. the new GUI already is a bit slower.

Sep 24, 2007 2:49 PM in response to xs4is

Would you prefer "workaround"? "Kludge"?

I'm glad to hear that L8 can use more than 4 gigs of ram in Leopard...but they still need to make that extra ram available to third party plugs.

Justin, I think it's significant that the use of this kind of thing in tiger hasn't been more widespread, and that it took apple this long to do it (frankly, I'm kind of surprised at their timing). Apple could have done this two and a half years ago, giving themselves a major advantage over pretty much every other audio app, but they didn't get it out until now. Apple has tried to convince their developers to use various tricks to access more memory in 10.4 (kinda sorta 64 bit), but few have gone to the trouble. If even apple is rarely willing to do it, how can they expect anyone else to?

Orren

Level 1

0 points

Sep 24, 2007 2:53 PM in response to xs4is

{quote:title=xs4is wrote:}
Second, the word "hackery" or any other similar word implies that it's broken or they're cheating somehow. This is just an incredibly well designed, well thought out way to access more RAM than a 32 bit Logic can. Calling the process "hackery" is kind of insulting.{quote}

I agree. That's why in my post, I used the word in quotes, to reflect Mike's denigration of the Logic coding team. I've made it abundantly clear that I consider these guys to be extremely clever—smarter than the average bear, to quote Yogi—and that what they've done definitely counts as brilliance, to my mind.

Orren

Sep 24, 2007 3:13 PM in response to Orren

I don't doubt that it took extreme cleverness to make it work, denigration is not my intention. But does it really make sense to expect extended memory support through each third party having to come up with an amazing brilliant breakthrough? Or just by documenting an update to the spec that they can all use (and are already using on the windows side in some cases)?

Insisting that doing it this way requires extra special savvy (and even the geniuses at apple took 2.5 years to release it) just makes it seem that much more ridiculous to insist that all third parties should do that instead of apple just releasing a 64 bit version.

Justin C

Level 6

8,700 points

Sep 24, 2007 3:31 PM in response to Mike Connelly

Mike Connelly wrote:
Would you prefer "workaround"? "Kludge"?

I'm glad to hear that L8 can use more than 4 gigs of ram in Leopard...but they still need to make that extra ram available to third party plugs.

Justin, I think it's significant that the use of this kind of thing in tiger hasn't been more widespread, and that it took apple this long to do it (frankly, I'm kind of surprised at their timing). Apple could have done this two and a half years ago, giving themselves a major advantage over pretty much every other audio app, but they didn't get it out until now. Apple has tried to convince their developers to use various tricks to access more memory in 10.4 (kinda sorta 64 bit), but few have gone to the trouble. If even apple is rarely willing to do it, how can they expect anyone else to?

I'm also surprised by the timing but we have to keep Logic's release cycle in mind. For all we know, this program (regarding EXS memory management) has existed for 2 years but other factors delayed Logic's release and the sensible time to introduce it did not happen until now. Development years ahead of time is not uncommon.

Regarding Apple 'rarely willing to do it' - maybe they didn't originally intend to but did based on demand. Again, we don't know. I really don't see 'rarely willing to do it' as the case -- 'have rarely had a real reason to do it' is more appropriate IMO. Again, I have not needed to develop a real program on this level so I can't speak firsthand, but it's typical in programming to work towards a usable solution with emerging technologies. If it doesn't work or will not work well enough, it doesn't get released. It's not really a matter of 'hacking', it's more appropriately measured by required man hours in order to create a usable implementation to the level the user base expects. There are really very few programs which have zero obstacles -- some obstacles are simply larger or require more time and creativity.

Finally, it will probably turn out that this implementation is better than the alternatives (secondary apps...) for end users. At that point, one really can't deny it's an implementation rather than a hack/workaround. Time will tell.

J

Sep 24, 2007 11:53 PM in response to Justin C

there are advantages and disadvantages to logic both going 64 bit or staying 32 bit. clearly they decided that the advatages in remaing 32-bit outweighed the disadvantages. parts of the program that require 64-bit processes have them, but where it doesn't matter it remains 32 bit. mike has been talking to people (East West most likely) who have been telling him that unless logic becomes a 64-bit application and 10.5 comes out 3rd party plugs won't be able to access more memory. but he is (and they are) wrong.

it can and is being done right now, by VSL and by the exs at least.

i have no idea why he is so insistent on this point (possibly because he has been convinced by whoever he has been contact with), but this has been thrashed out already on another thread.

one thing he is right about is that 'eventually' logic will become 64 bit, but i would have thought that won't happen until v9 at the earliest. (FWIW, yes, that IS speculation)

zibba

Level 2

255 points

Sep 25, 2007 5:29 AM in response to Mike Connelly

Mike Connelly wrote:
So do they access more than 4 gigs of memory?

I've written a couple of AU's for fun, not for profit. It's hard to answer this question in the specifics because we're not able to see the exact nature in which any particular AU is implemented, only the developer's know.

For the vast majority of AU's the answer would be no. Effect and instrument plug-ins, in general, have fairly modest memory requirements. Typically, it's only when you start looking into samplers that the memory requirements can become more intense as the size of the sample libraries increase.

That said, I think the confusion surrounding EXS24 comes largely from past experiences with previous versions of Logic, it's use of terms like "virtual memory", "streaming" and perhaps some confusing claim about it using the fad term 64-bit when necessary.

Regardless of that, as I stated elsewhere, it is possible to access all of physical memory through a technique known as mmap - which maps files into memory.

Now I know this forum is based upon a weak trust system, I don't know any of you, you don't know me, so, that said, and without getting into a major computer science lesson, because this forum is for musicians, here's how it works in layman's terms.

The first concept to grasp is known as virtual memory. This extends the size of the computer's RAM to be as big as the hard drive. Typically, if you have 2GB of RAM, you can launch many more applications than would fit into that 2GB. This is because the operating system is swapping small pieces of memory between the RAM and the hard drive.

When the operating system opens a file and the contents of that file get read from the disk into RAM, the operating system attempts to hold on to that file for as long as possible. This is called file caching. It enables many programs accessing the file to quickly retrieve the contents without having to re-read it from the slow hard drive every time.

By combining these two schemes, it's possible to map a file into memory and access it's data directly. This is known as memory mapping (mmap).

Now for the confusing part. A 32-bit application can access only 4GB of RAM. Also, in a 32-bit computer you can only have a maximum of 4GB of RAM. But the G5 and Intel Mac's are 64-bit computers. So you can have more than 4GB of RAM.

The applications that we write today are 32-bit. Therefore, an individual application can only access 4GB of RAM. By "application" I mean, application or plug-in or anything running outside of the operating system internals.

However, that does not mean that the operating system has to be 32-bit. In the case of OS X, it is in fact 64-bit. Therefore, the operating system can access RAM larger than 4GB, but the applications cannot.

So, by using the 2 schemes of virtual memory and file caching, combined with the mapping of files into memory, the operating system is able to load those files into places that the application cannot ordinarily access, map it, and provide a means for that application to read that data.

This is done by the application referring to the file by an identifier which the operating system converts (maps) to a 64-bit address, thus reading the data and passing it back to the application.

Finally, because the operating system is 64-bit, it can also read files larger than 4GB. This is also made available to the application through the use of a 64-bit file offset, so the application can access any part of the larger file.

I believe that it is this last point that has caused the confusion over whether it is, or is not 64-bit.

Simple isn't it?

Sep 25, 2007 5:29 AM in response to Rohan Stevenson1

"clearly they decided that the advatages in remaing 32-bit outweighed the disadvantages"

Huh?

They don't even have a fully 64 bit OS shipping yet. If they shipped a fully 64 bit version today, it wouldn't even be able to run on 10.4. Sure, it may be a while before they ship 64 bit logic, but interpreting the fact that they didn't ship a 64 bit version when the OS doesn't yet support it as "clearly deciding" that 32 bit is better or something is just ridiculous.

Sure, there are ways to find loopholes around the current limits, but 64 bit support is a better and more standard way to do it. Most developers who want to support extended ram will do it with 64 bits instead of trying to jump through hoops.

VSL is running their plugin as a separate app. It works for them, but the idea of a bunch of different plugins doing this in a bunch of different ways makes me paranoid. I'd much rather see 64 bit support, and plugins running as plugins.

I'm not sure why you're so insistent that apple shouldn't go 64 bit. Once they do, if they don't want to drop support for a number of machines (including some current ones) they'll be releasing a version that can run in 32 bit mode as well, most likely as a fat binary that contains both sets of code. I wouldn't be surprised if 64 bit apps will be able to run in 32 bit mode on 64 bit hardware by setting a pref somewhere similar to rosetta. Or if you're really scared of 64 bits, you can just stay with 10.4.

Sep 25, 2007 5:30 AM in response to zibba

bravo zibba. brilliant explanation.

Sep 25, 2007 7:01 AM in response to zibba

hi zibba,
thank you for taking the time to explain this.
anytime you are in Sydney, i'll buy you a beer.
DR9.

Sep 25, 2007 7:32 AM in response to Mike Connelly

They don't even have a fully 64 bit OS shipping yet. If they shipped a fully 64 bit version today, it wouldn't even be able to run on 10.4.

sigh. read zibbas explanation. it pretty well spot on.

I'm not sure why you're so insistent that apple shouldn't go 64 bit.

because there are performance disadvantages as well as the fact that 32-bit plugs (that have not been updated) will not work. i don't know why that is so don't ask me.

logic runs 64-bit where necessary and 32-bit where it doesn't matter.

10.4 is 64-bit as well. leopard comes with other optimisations which we should benefit from such as faster GUI and some other fancy stuff i didn't take the trouble of understanding.

you need to ask yourself why the developers of logic took this decision. clearly, since they were virtually rebuilding the program from the ground up it would have been discussed. the pros and cons would have been weighed up and they reached the decision that they did because it was the best one. the only reason to go 64-bit is because of the memory access and as has been demonstrated, explained, thrashed out and mangled here it isn't necessary to it that way for the one case system memory is important for what we do - sampler instruments.

Sep 25, 2007 8:47 AM in response to Rexy

A few random comments:

- VSL is not able to access more than 32 bits' worth of RAM right now. But the AU plug-in you open is just an audio/MIDI/router-streamer and remote window-opener for a program running outside Logic, and therefore its RAM access is independent of Logic's.

- You can run multiple stand-alone Vienna Instruments players, each with their own 3.5GB or so of RAM access, by copying the program and renaming each copy. The problem is that the V.I. Player doesn't access multiple audio interface outputs, so you can only get one stereo mix of whatever you're running.

- VSL has announced a new stand-alone mixer/host for their instruments, called Vienna Ensemble. Presumably it will solve this and the RAM issue.

- Mike Connely wrote: "VSL is running their plugin as a separate app. It works for them, but the idea of a bunch of different plugins doing this in a bunch of different ways makes me paranoid. I'd much rather see 64 bit support, and plugins running as plugins."

And I'd much rather see plug-ins running as stand-alone programs outside the sequencer so you don't have to load them with every Project and reload them when Logic crashes (no dis intended, just talking about reality: programs crash). The Core Audio standard seems to do a good job of avoiding conflicts right now, so I don't see why that would be a problem in the future. Today Soundflower, a free virtual audio interface from www.cycling74.com, can route things around between programs just fine.

Message was edited by: derivativemusic

Sep 25, 2007 8:50 AM in response to Rohan Stevenson1

I don't disagree with zibba's explanation. You just need to be aware that 32 bit apps can't use 64 bit memory directly.

1 A 32 bit app can only access 4 gigs of ram directly.
2 A 64 bit app can access more than 4 gigs.
3 10.4 only supports 64 bit apps without GUI (it's only PARTIALLY 64 bit, not accurate to call it a 64 bit OS)
4 In 10.4 a GUI app can access more than 4 gigs by creating a separate 64 bit process or app and sending data back and forth
5 In 10.5 a GUI app can run as a 64 bit app and use more than 4 gigs directly.

Make sense?

Performance issues are a valid concern. If you're worried about them and don't care about using more ram, just run the 32 bit version, or even stick with 10.4 so you don't get any of that nasty 64 bit stuff on your machine.

But that's no reason for them not to update, particularly if they ship it as a fat binary with both 64 and 32 bit versions (which they will have to do if they don't want to cut off support to many machines).

"you need to ask yourself why the developers of logic took this decision."

What decision? Not to ship the half of a fat binary that the public can't use for another month?

At this point we have no idea if Apple has a 64 bit version ready or not. We only know that they haven't shipped one yet. Again, you're making assumptions and pretending to know what apple is thinking.

Rohan, assuming Apple ships a 64 bit version at one point, and assuming it was a fat binary that allowed you to run it in 32 bit mode if you are worried about performance or compatibility issues, would you still have an objection to them doing a 64 bit update?

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

EXS in L8 versus L7: no more 3.5 GB RAM limit (with screen grabs)