The Portal FPS Experiment 2.0

[DrLex]

Here's yet another attempt at comparing frame rates between systems. It's basically the same experiment as the previous one (http://forums.steampowered.com/forums/showthread.php?t=1270589), but with the bugs corrected and some extra tests. This time, there's a demo with visible fog! This is important, as it might help Valve to investigate why fog slows down many Macs.

We'll be using a few ‘timedemos’ that play back some scenes from the game. This ensures that everyone runs the same tests. Don't worry if you haven't finished the game yet, the first 5 demos are short and give no spoilers.
Download the timedemos (http://homes.esat.kuleuven.be/~athomas/tmp/Portal-Timedemos2.zip) and unzip them. Drop the .dem files in ~/Documents/Steam Content/yourSteamID/portal/portal/

Close all CPU-intensive programs and browser windows with flashy flash content before starting Portal. Make sure the console is enabled in the game settings, at ‘Keyboard’ -> ‘Advanced’. In the video settings, disable vertical sync.
Bring up the console with the ‘`’ or ‘~’ key. Run the demos by entering “timedemo view1”, “timedemo view2b”, etc. Run each demo at least twice to reduce effects of caching. Demo 'view2b' should be run at least three times, preferably more, due to the randomness of the fog. Report the highest fps value for each demo, together with your computer and graphics card model, and any special video settings.

If you wish, you can also run a much longer demo, made by jeanlain, to get an indication of your average framerate during normal gameplay. Beware though that this demo contains spoilers for a crucial part in the game, so if you haven't finished Portal yet, you may not want to run this demo, or close your eyes while it runs ;)
Download jeanlain's demo here (http://public.me.com/selene.prod). Run it with “timedemo mydemo1”.

Feel free to report multiple results for different graphics settings, and results under Boot Camp. The previous experiment has already revealed that anti-aliasing (AA) severely reduces the framerate on most Macs. It's better to set video to native resolution with no AA, than to use a lower resolution with AA. If you want to experiment with AA, first do all tests without AA.
If you did the previous experiment, you can copy your values from the old thread (http://forums.steampowered.com/forums/showthread.php?t=1270589) and only need to (re-)do 'view2b', 'view5' and the longer test if you wish.


Here are results for my MacBook Pro 2.33GHz Core 2 Duo (2006), ATI X1600, which is about the ‘minimum requirements’ system. Resolution is 1024x640 and settings are the recommended ones, except textures and shader are both 'high', one notch above recommended.
view1: 26.71fps
view2b: 24.18fps
view3: 32.67fps
view4: 38.95fps
view5: 10.55fps
jeanlain's demo: 32.74fps

[GangztaX]

Old results :

Now this is curious.
My machine :
Intel Dual Core 2.40 ghz
3GB ram
2GB HD5770
Win7/SnowLeopard.


Windows 7 64bit :
Resolution : 1920x1200
Model Detail: High
Texture Detail: Very High
Shader Detail: High
Water Detail: Reflect All
Shadow Detail: High
Color Correction: Enabled
Antialiasing: Max
Filtering: Max
Motion Blur: Enabled
FOV : Max
Vsync DISABLED

View1: 245.19
View2: 199.52
View3: 213.60
View4: 259.88


OSX SNOW LEOPARD :

The settings I used are identical to the Windows 7 ones.

View1: 97.91
View2: 75.40
View3: 87.23
View4: 101.92


New results :

Win7 :

View1: 224.19
View2: 189.22
View3: 233.79
View4: 248.41
View5: 191.98

Mac OSX :

View1: 89.12
View2: 81.31
View3: 82.73
View4: 98.01
View5: 67.92 <---- wtf ?

[CC>>skippy]

lol, i average about 30-40 FPS with bilinear filtering, no Vsync or color correction, high texture, and everything else is preset to "recommended"

[DrLex]

View5: 67.92 <---- wtf ?
That's perfectly normal, it is the most FPS-killing scene in the game :)

[dreyesceron]

I just ran the Tests on My Mac. Here are my results...

iMac 7,1 (2007)
10.5.8 Leopard
2 Ghz Core 2 Duo
Memory: 2 GB
ATI Radeon HD 2400

Res: 1024 x 640

All settings set at Highest (Filtering Mode: Anisotropic 16x)
Portal Render Depth: 2
Color Correction, Antialiasing, VSync, Motion Blur Disabled
Field of View: 90

Demo 1:
Test 1: 29.01 FPS (34.47 ms/f) 12.618 Variability
Test 2: 28.93 FPS (34.57 ms/f) 13.896 Variability

Demo 2b:
Test 1: 16.50 FPS (60.60 ms/f) 11.371 Variability
Test 2: 16.71 FPS (59.83 ms/f) 10.618 Variability
Test 3: 17.10 FPS (58.49 ms/f) 11.605 Variability

Demo 3:
Test 1: 36.12 FPS (27.69 ms/f) 9.665 Variability
Test 2: 36.04 FPS (27.75 ms/f) 9.701 Variability

Demo 4:
Test 1: 40.63 FPS (24.61 ms/f) 15.506 Variability
Test 2: 43.59 FPS (22.64 ms/f) 10.354 Variability

Demo 5:
Test 1: 8.79 FPS (113.81 ms/f) 1.636 Variability
Test 2: 8.35 FPS (119.70 ms/f) 2.428 Variability

UPDATE:

Same Tests on the Same Machine running Boot Camp running Windows XP

Resolution: 1024 x 600

All other settings are the same as the Mac OS X Tests

Demo 1:
Test 1: 42.51 FPS (23.52 ms/f) 38.928 Variability
Test 2: 43.01 FPS (23.25 ms/f) 44.068 Variability

Demo 2b:
Test 1: 31.31 FPS (31.94 ms/f) 38.853 Variability
Test 2: 31.29 FPS (31.96 ms/f) 47.194 Variability
Test 3: 32.38 FPS (30.88 ms/f) 47.189 Variability

Demo 3:
Test 1: 53.57 FPS (18.67 ms/f) 46.463 Variability
Test 2: 52.29 FPS (19.13 ms/f) 46.625 Variability

Demo 4:
Test 1: 63.92 FPS (15.65 ms/f) 67.697 Variability
Test 2: 65.11 FPS (15.36 ms/f) 70.351 Variability

Demo 5:
Test 1:18.57 FPS (53.85 ms/f) 15.862 Variability
Test 2: 19.35 FPS (51.68 ms/f) 13.337 Variability


So running these tests, the differences in FPS are pretty large. Demo 1 had about a 12 Frame Difference, Demo 2b had 15 difference, Demo 3 a 17, Demo 4 a 20 and Demo 5 an 11 frame difference. Windows obviously performed much better than Leopard in this case.

[imacken]

OK,
1: 119 (AAx2) 179 (noAA)
2b: 132 184
3: 156 196
4: 176 258
5: 68 93

This is on Snow 10.6.3, i7 920 @ 4GHz, GTX285 1Gb, 6Gb DDR3 and 1920x1200.
All settings were maxed except filtering (Ani x2) and AA as above.

[ClassicGOD]

Before you compare FPS between Windows and Mac versions remember that Full HDR is ON at all times in Mac version and theres no way to turn it off (mat_hdrlevel 0 just goes to loading for a few seconds and resets back to 2). At least thats the case on my MacBook Pro.

[thedreadedgman]

Ok here are my results from many runs of testing.
Scaling appears to be about 70% of Windows speed, best scaling is about 85%, but only with view5...

Test setup:
iMac Late 2009 (C2D 3.06Ghz, 4GiB RAM, ATI Radeon HD 4670, 21.5" 1920x1080 screen)

-- Native: Mac OS X v10.6.3
-- Bootcamp: Windows 7 x64 with Radeon 10.4 drivers

2 of each demo, 3 of view2b, highest of each group, round down to nearest whole FPS

Test 1 Resolution: 1280x720
Portal Depth: 2

Graphics Settings:
Model detail: High
Textures: High
Shaders: High
Water: Reflect All
Shadows: High
Color correction: Disabled
AA: None
AF: 4X
VSync: Off
Motion Blur: Off

Windows only options:
Multicore: Disabled
HDR: Full
Bloom: Not Checked
demoname: windows, mac
view1: 174, 111
view2b: 134, 101
view3: 189, 118
view4: 241, 153
view5: 57, 49
jeanlain: 140, 96 (mydemo1.dem file)


---
Test 2 Resolution: 1920x1080 (native)
Portal Depth: 2

Graphics Settings:
Model detail: High
Textures: Medium
Shaders: High
Water: Reflect All
Shadows: High
Color correction: Enabled
AA: None (also 2x tested)
AF: 2X
VSync: Off
Motion Blur: Off

Windows only options:
Multicore: Disabled
HDR: Full
Bloom: Not Checked
demoname: windows, mac, mac 2xAA
view1: 82, 58, 22
view2b: 63, 52, 23
view3: 102, 78, 30
view4: 112, 84, 30
view5: 29, 26, 11
jeanlain: 96, 70, 26 (mydemo1.dem file)

[ClassicGOD]

Bloom should also be enabled on Windows.

The differences are more or less what I expected - Mac OS X average FPS are lower but it gets less of a hit from resolution. Unfortunately Mac OS X is more overall performance optimized than gaming performance optimized, you can open Dashboard or Expose when running Portal without any problem and that philosophy provides huge blow to the number of fps. Unless Apple implements some kind of 'Gaming mode' when running fullscreen OpenGl apps (not to mention they could do better job with drivers :) ) Mac OS X will be slower than Windows in games. Valve still has some optimization to do - they should be able to come to about 10-15% difference between Mac and Windows.

[mitch_mac]

Q: Whats demofile is for jeanlain: 96, 70, 26 ?
Is that the longrun mydemo1.dem ? If yes perhaps better to use that name than jeanlain ?!
Or jeanlain renames his dl file to jeanlain.dem ?
Or perhaps add that name beside the dl link.
THANKS for great FPS Informations at first posting - very good undertandable :)

[MachineShedFred]

MacPro 4,1 (early 2009) @ 2.66 Ghz - GTX 285
Using:

1024x640, portal depth =2, everything maxed except AA, using bilinear.

view1: 118.57
view2b: 151.07
view3: 120.41
view4: 184.58
view5: 67.60
mydemo1: 100.73

For grins, here's the numbers for 1920x1200, all other setting equal.

view1: 60.35
view2b: 68.43
view3: 72.88
view4: 92.43
view5: 33.40 :o
mydemo1: 67.04

Only other apps running were Mail, and Chrome for writing this stuff into. No flash or any of that crap running in Chrome; though CPU is totally not the bottleneck here - hl2_osx doesn't even use 100% of one core.

[Grom_Reaper]

iMac7,1 (Mid 2007)
Intel Core 2 Duo, 2.4 GHz
4 GB RAM
ATI Radeon HD 2600

Settings: VSync and Colour Correction disabled, everything else high/very high, no AA, trilinear filtering at native resolution.

View 1: 26.17 fps (38.21 ms/f) 14.044 fps variability
View 2b: 17.38 fps (57.55 ms/f) 7.459 fps variability
View 3: 33.01 fps (30.29 ms/f) 9.428 fps variability
View 4: 38.45 fps (26.01 ms/f) 13.477 fps variability
View 5: 10.62 fps (94.15 ms/f) 8.146 fps variability
Jean:30.04 fps (33.29 ms/f) 11.832 fps variability

[pokrface]

27" Core i7 iMac (11,1), 4GB RAM, Radeon 4850 (Mobility, in spite of what Apple says)

Resolution: 2560x1440 (native)
Full screen
Model detail: high
Texture detail: very high
Shader detail: high
Water detail: reflect all
Shadow detail: medium
Color correction: disabled
AA: none
AF: 8x
VSync: off
Motion blur: disabled
FoV: 75

view1
First run: 58.57 fps, 5.122 fps variability
Second run: 57.95 fps, 4.125 fps variability

view2
First run: 89.73 fps, 3.064 fps variability
Second run: 91.13 fps, 2.716 fps variability

view2b
First run: 50.04 fps, 3.735 fps variability
Second run: 52.90 fps, 3.064 fps variability

view3
First run: 78.33 fps, 3.158 fps variability
Second run: 78.39 fps, 2.718 fps variability

view4
First run: 88.02 fps, 4.924 fps variability
Second run: 88.60 fps, 2.452 fps variability

view5
First run: 29.35 fps, 8.323 fps variability
Second run: 30.62 fps, 9.308 fps variability

jeanlain's demo
First run: 71.33 fps, 8.988 fps variability
Second run: 73.36 fps, 7.003 fps variability



edited to add--I didn't bother quitting any background apps, so this is with iTunes, Skype, Adium, Firefox, Dropbox, and lots of other little things still running. With significant chunk of RAM still free and the CPU so heavily underutilized by Portal (whole cores sitting idle), the presence of those other apps should make little-to-no difference.

[LTCHIPS]

Bloom should also be enabled on Windows.

The differences are more or less what I expected - Mac OS X average FPS are lower but it gets less of a hit from resolution. Unfortunately Mac OS X is more overall performance optimized than gaming performance optimized, you can open Dashboard or Expose when running Portal without any problem and that philosophy provides huge blow to the number of fps. Unless Apple implements some kind of 'Gaming mode' when running fullscreen OpenGl apps (not to mention they could do better job with drivers :) ) Mac OS X will be slower than Windows in games. Valve still has some optimization to do - they should be able to come to about 10-15% difference between Mac and Windows.

Have you guys tried lunch options yet? Or are those only compatible with Windows?

[Grom_Reaper]

Have you guys tried lunch options yet? Or are those only compatible with Windows?

I love toasted ham & cheese sandwiches, but Portal seems to be lacking in them for the Mac :(

[Mangr0v3]

No mushroom or avocado? Leave my sight at once!

[rbarris]

The differences are more or less what I expected - Mac OS X average FPS are lower but it gets less of a hit from resolution. Unfortunately Mac OS X is more overall performance optimized than gaming performance optimized, you can open Dashboard or Expose when running Portal without any problem and that philosophy provides huge blow to the number of fps.

Snow Leopard might be more clever than you think in this area. When the game is full screen and nothing is popped in front of it, we're getting the efficiency of page flipping, not the composited window path. But it is able to allow other windows (speaker volume popups, growls, etc) to come to the front - it switches on the fly to composited mode and back when those things come and go. So having the flexibility in itself isn't placing a permanent drain on the performance.

The issues in getting peak performance out of Source engine on the Mac are more subtle and varied, but as I said it's something we will continue to dig through. There's a steady traffic flow between all the relevant parties here, and we've already found a number of tune-ups that are going to get done. We will push the cart some more.

[demigod_mac]

The things rbarris mentioned I absolutely love about the Mac versions.

It's so annoying on Windows to tab out of fullscreen Source games, only to have them seize up for ~5 or so seconds of black screen when tabbing back in, whereas on OS X you can tab through them in a fraction of a second. And all the other things like Growl messages can superimpose themselves over the game window (also convenient).

rbarris - any chance of adding Growl support to Steam? Or at least, offering it as an option for notifications instead of the default ones.

Also curious why such a disparity in performance between OS X and Windows... are there a lot of small factors too numerous to mention, or some big "derp" factor where Apple could magically add 10+ fps by flipping a line of code?

[Mangr0v3]

It's so annoying on Windows to tab out of fullscreen Source games, only to have them seize up for ~5 or so seconds of black screen when tabbing back in

Only 5 seconds? It's 20-50 for me :(

[imacken]

Guys, I already posted some of the following, but I've added my Windows 7 results:
1: 119 (10.6.3 AAx2) 179 (10.6.3 noAA) 169 (W7 AAx2) 207 (W7 noAA)
2b: 132 184 154 171
3: 156 196 199 238
4: 176 258 235 286
5: 68 93 81 93

This is on i7 920 @ 4GHz, GTX285 1Gb, 6Gb DDR3 and 1920x1200.
All settings were maxed (incl HDR) except filtering (Ani x2) and AA as above.

This shows that W7 has better frame rates, but maybe not as much as some of you might think.

[ClassicGOD]

Snow Leopard might be more clever than you think in this area. When the game is full screen and nothing is popped in front of it, we're getting the efficiency of page flipping, not the composited window path. But it is able to allow other windows (speaker volume popups, growls, etc) to come to the front - it switches on the fly to composited mode and back when those things come and go. So having the flexibility in itself isn't placing a permanent drain on the performance.

The issues in getting peak performance out of Source engine on the Mac are more subtle and varied, but as I said it's something we will continue to dig through. There's a steady traffic flow between all the relevant parties here, and we've already found a number of tune-ups that are going to get done. We will push the cart some more.Sweet, I didn't knew it was doing that. Anyway I'm just beginning to program for Macs (and I'm starting on the iPhone :D ) so I'm not going even to pretend that I know even 1% of what you know. So thanks for the info :cool:

[jeanlain]

Guys, I already posted some of the following, but I've added my Windows 7 results:
1: 119 (10.6.3 AAx2) 179 (10.6.3 noAA) 169 (W7 AAx2) 207 (W7 noAA)
2b: 132 184 154 171
3: 156 196 199 238
4: 176 258 235 286
5: 68 93 81 93

This is on i7 920 @ 4GHz, GTX285 1Gb, 6Gb DDR3 and 1920x1200.
All settings were maxed (incl HDR) except filtering (Ani x2) and AA as above.

This shows that W7 has better frame rates, but maybe not as much as some of you might think.
Interesting... Anandtech used the same GPU and got much lower fps under OS X compared to win 7. AA does have a higher effect on performance under OS X.
Are the settings exactly the same between OS X and win 7 and are drivers up-to-date?

[rbarris]

Anandtech wasn't running the same class of test.. Timedemo vs steady view of a scene right?

[imacken]

Are the settings exactly the same between OS X and win 7 and are drivers up-to-date?
Yes to both.

[dirk0gently]

Yes to both.

but they are not the same drivers. You are using two different rendering systems as well. You cannot expect them to act exactly the same.

[imacken]

but they are not the same drivers. You are using two different rendering systems as well. You cannot expect them to act exactly the same.
Eh, nobody said they should!
Have you read this thread? It is mainly about comparing OS X and Windows framerates in Portal.

[jeanlain]

Anandtech wasn't running the same class of test.. Timedemo vs steady view of a scene right?
But the steady views use the timedemo command here. Other users here report larger differences between OS X and Window, not only in the demo I uploaded.

[dirk0gently]

Eh, nobody said they should!
Have you read this thread? It is mainly about comparing OS X and Windows framerates in Portal.

I can understand that, but you have no baseline for comparative readings. I can understand if both were using the same rendering system, eg OpenGL or DX, but the difference in speed and performance on each system is not relative to anything. Sure you can compare them but I don't see a solid baseline for comparing both systems.

[jeanlain]

Old results :




New results :

Win7 :

View1: 224.19
View2: 189.22
View3: 233.79
View4: 248.41
View5: 191.98

Mac OSX :

View1: 89.12
View2: 81.31
View3: 82.73
View4: 98.01
View5: 67.92 <---- wtf ?
OS X doesn't ship with drivers for radeonHD 5 series AFAIK, so the big difference between OS X and Windows is hardly surprising. I wonder how you managed to get any 3D acceleration at all with your radeonHD 5770 under OS X.

[DrLex]

I love toasted ham & cheese sandwiches, but Portal seems to be lacking in them for the Mac :(
I guess we'll have to wait for TF2. Om nom nom nom...

[imacken]

I can understand that, but you have no baseline for comparative readings. I can understand if both were using the same rendering system, eg OpenGL or DX, but the difference in speed and performance on each system is not relative to anything. Sure you can compare them but I don't see a solid baseline for comparing both systems.

Fair enough, but I don't think anyone's trying to be scientific about these tests. They are purely for general interest comparing the end results to the game player on Windows and OS X.

[jeanlain]

I can understand that, but you have no baseline for comparative readings. I can understand if both were using the same rendering system, eg OpenGL or DX, but the difference in speed and performance on each system is not relative to anything. Sure you can compare them but I don't see a solid baseline for comparing both systems.
It shows that one software takes better advantage of the hardware (such comparison was not possible before intel macs).

[dirk0gently]

It shows that one software taker better advantage of the hardware (such comparison was not possible before intel macs).

Yeah sorry, I was reading the posts wrong.

[multigl]

OS X doesn't ship with drivers for radeonHD 5 series AFAIK, so the big difference between OS X and Windows is hardly surprising. I wonder how you managed to get any 3D acceleration at all with your radeonHD 5770 under OS X.

I too would like to know how this was accomplished.

[Mangr0v3]

People have been flashing non-Mac-specific video cards for years.

[imacken]

Or he's on a hack like lots of others here.

[jeanlain]

People have been flashing non-Mac-specific video cards for years.
Hackintoshes just use PC cards with their original BIOS. There's no Mac EFI available to flash a radeonHD 5 card with anyway. But you need appropriate drivers to get video acceleration (including Quartz Extreme and Core Image). I have never heard of a tweak that enables acceleration on a generation of GPU that doesn't ship with Macs. Maybe GangztaX tweaked the radeonHD 4xxx kext so that they work with the 5770, but that not panacea.

[Draeconis]

I'm sorry, this does sound like a great idea, but if you're doing an experiment, you limit your environmental variables as much as possible to get the most from it, so surely it should involve specific video settings?

It's great that a lot of people want to get involved, myself included, but if everyone's running it at differing resolutions with different graphics options on/off/higher/lower, doesn't that make it a lot harder to tell what's causing different results between machines (trying to account for hardware performance differences, of course).

[DrLex]

I'm sorry, this does sound like a great idea, but if you're doing an experiment, you limit your environmental variables as much as possible to get the most from it, so surely it should involve specific video settings?

It's great that a lot of people want to get involved, myself included, but if everyone's running it at differing resolutions with different graphics options on/off/higher/lower, doesn't that make it a lot harder to tell what's causing different results between machines (trying to account for hardware performance differences, of course).
This is not intended to be a rigorous experiment to publish exact speed differences between Macs in a scientific paper. There are better benchmark programs for that. It's just to get an idea what framerates can be obtained on different models with different video settings. I'm not even sure if we could find a meaningful set of video settings that can be set on all Macs. That's why I recommend that people run multiple tests with different settings, so that we can see how certain models are affected by those settings.

This thread was a response to another thread that was a lot worse. It just asked “post your FPS here” without any constraints. People didn't even mention in what part of the game they measured FPS (and how) or with which video settings.

[MaacK]

27" Core i7 iMac (11,1), 4GB RAM, Radeon 4850 (Mobility, in spite of what Apple says)


Apple actually puts mobility video cards in desktop computers? Wow, I officially hate Apple/Mac even more.

[Bolteh]

Apple actually puts mobility video cards in desktop computers? Wow, I officially hate Apple/Mac even more.

Then what the poop are you doing here, reading a Mac forum?

[jeanlain]

Apple actually puts mobility video cards in desktop computers? Wow, I officially hate Apple/Mac even more.
Apple cannot really use desktop GPUs in a thin all-in-one desktop that is supposed to be silent. The heat sink of the desktop 4850 would not even fit there.
But they should indicate that they use mobility GPUs in the tech specs.