Virtual Drive no longer virtual

Atari takes another step closer :: June 23rd, 2006 :: Steam updates :: 18 Responses

Steam’s virtual drive has long been known to erode the loading performance of games using it. A new filesystem beta started last night containing Valve’s solution: accepting the inevitable and leaving GCFs for the full system.

Games using what was formerly the Steam virtual drive now have their files loose on the hard drive in /common, a subfolder of /SteamApps. Small NCF indexes take the place of GCFs to ensure that the standard filesystem’s features (alternate content mapping, “the various Steam APIs” and streaming) can still operate.

John Cook explains:

Any virtual file driver causes an extra cross-process synchronization to occur, which although small by itself can add up to a lot of time when you have tens of thousands of read requests. The quickest thing that can be done is simply not doing that work. Going forward we have two strategies for SteamFS:

Direct SteamAPI file access (GCF), which lets the app take advantage of the cache file format’s strengths (lots of small, easily updatable files in a compact format).

Loose files on disk (NCF), which lets old or licensed games run on Steam without integration work or any unnecessary indirections.

Once we realized that we could still support our Steam features while having loose files on disk, it became an easy choice to do it for old games.

The changes bring the GamersFirst programme, overshadowing so many of Steam’s public developments recently, one step closer to availability.

The virtual drive’s demise also opens up WINE/Cedega emulation for games that do not implement the standard system.

Also, please note that the standard GCF implementation does not suffer from the latency of the virtual drive. The virtual drive is an extra layer above the Windows filesystem; the full GCF implementation replaces it.

Tags: John Cook, No Cache File, performance

18 Responses to this post:

Comments

June 23rd, 2006 at 10:54 pm ::
DiSTuRbEd Says:

I was wondering when you were going to post about this. 😉 Saw your post on Ship beta forums aswell, think you will make a future update about that sometime sooner or later?
June 23rd, 2006 at 11:05 pm ::
Tom Edwards Says:

Yes, of course. 🙂 It’s one of the earliest announced functions of Steam 3.0.
June 23rd, 2006 at 11:28 pm ::
DiSTuRbEd Says:

I can’t wait to see Auto-Updates working on Steam, no more restarting Steam to recieve news/updates=WIN.
June 24th, 2006 at 12:07 am ::
ATimson Says:

I hesitate to call this a solution. I still think that they would’ve been better served by optimizing the driver, rather than dumping it entirely. (Of course, the strengths of the GCF that John pushed are pretty much the antithesis of the Unreal engine format files as well as the data files used by Earth 2160 and Shadowgrounds, so I can see why they might not want to bother with the current crop of games.)

Is this going to mean that users will have two copies of the game on disk (like Darwinia and Codename: Gordon, both pre-SteamDrv.sys), one in the cache file and one expanded? Or will there be no cache file for games that use the NCF format? And is Steam smart enough to delete the cache file after unpacking, in the latter case?

(The filesystem beta doesn’t work with Shadowgrounds—I haven’t tried Earth 2160 yet—which is why I don’t know. :))
June 24th, 2006 at 3:05 am ::
Andy Simpson Says:

If you go across a process boundary you get overhead. It’s just inevitable, there’s no way round it. This really is just the most sensible way to go, keeping this overhead to the absolute minimum. I’m glad Valve aren’t too pig-headed to stick to GCF when it’s just not working very well in some situations. Bravo.

I’d guess since the NCF thing is going to replace the GCFs for those games we’re not going to get the same kind of wasteful behaviour.

Hmm. Maybe we’ll start seeing even with proper GCF Steam games that the bits which have to be expanded out right now to the filesystem will be kept out of the GCF caches and kept in NCF?
June 24th, 2006 at 8:43 am ::
Tom Edwards Says:

Is this going to mean that users will have two copies of the game on disk (like Darwinia and Codename: Gordon, both pre-SteamDrv.sys), one in the cache file and one expanded? Or will there be no cache file for games that use the NCF format?

There will be no cache files for No Cache Files. 😉
June 24th, 2006 at 9:25 pm ::
ATimson Says:

Obviously, they won’t be used during gameplay. But there’s nothing to keep Steam from using them for download purposes as their current NCF games do. :p
June 25th, 2006 at 9:27 am ::
Tom Edwards Says:

Sure there is: hard drive space. Having the GCFs and /common doubles up the requirements. The NCF games of today only have their GCFs remaining in case the user leaves the beta.
June 25th, 2006 at 12:51 pm ::
DiSTuRbEd Says:

So I am guessing once the beta is done(this is for people that didn’t use the beta of course), the GCF will be extracted and you will have to delete it?
June 26th, 2006 at 8:16 am ::
Tom Edwards Says:

If you own RO, yes.
July 13th, 2006 at 2:43 am ::
Jobye Says:

Shadowgrounds has just now made the move to the new filesystem as well.

The Steam Review