PDA

View Full Version : The source engine - a success (fyi)


Boff
03-13-2010, 03:24 PM
It feels like some days, there are only a handfull who champion the source engine, and with the new pictures of Portal2 floating around, it looks like valve have really put time and effort into eye candy and the rendering on the 2010 build of the engine.

But I consider Source, and I'm sure valve consider it, a success.

And this is a little For Your Information, for those who was not with, or just plainly forgotten the concept of Source.


Source : A Strategy and A concept

This is why I think source is a success, because you have to remember how and why the source engine EXISTS.

Valve created the source engine, to make HL2, but also to address and issue it had noticed that plagued the gaming industry.

Much in the same way they had problems with patching GldSRC engine for HL1/CS/TFC->TFC1.5 they envisioned STEAM, where they could distribute Patches, and act as DRM, and it not very popular from the begining, I think we can agree that steam as a strategy for valve, has been a success.

And thusly we come back to Source being a success, as it too was a strategy.

Not only did valve have to build and then patch for the old engine, but they had to develop the next generation engine and hit all the technical milemarks at the release.

And then after that, they had to build the 3rd engine, whilst patching engine number 2, for expansions and what not.

But valve opted NOT to do this traditional engine building.

SOURCE - a Brand name
Source got it's own funky logo, and it was going to take on multiple-personality syndromes with the following concepts, the brand-name would tie it all together so everything would be OKAY, or so valve very simarily thought about about the naming convention of the episodes......

CONCEPT 1a
What IF Valve created an engine, and ALL the games built on that engine would run on the same engine,

so a patch would only have be run out "once" to fix the engine via steam.

CONCEPT 2
IF the engine was built in a modular fashion, and theoretically, the modules would be replaced in time, so the OLD renderer get replaced with a more modern renderer that addresses the hardware at that future. The network code could be replaced, the sound system etc etc, until all the modules have been replaced, and suddenly you have a iterated into a NEW engine.
http://www.youtube.com/watch?v=bk24RdfXWcg

(oddly enough people still have problems with this concept, and still compare The source engine that powers HL2(2004/5) with games from 2009)

CONCEPT 1b Idealism - concept 1 revisited
Now that the all the source engine games used the same engine and would always be patched then thusly all the engine would always be upgraded, and of cousrse then all your source engine based games would be STATE OF THE ART since they ran of the same build of the engine.

You would also avoid the unpleasant scenario's of these shoe-horning in concepts into a engine that can not or barely can handle them.
If anyone remembers the APC's and other vehicle experiments in the original CounterStrike, as well as the map overlays and gui-systems that eventualy slipped into TFC 1.5.


SUCCESS AND FAILURE
As of 2005 concept 1b failed with The lost Coast.
As valve iterated toward the first major milestone of change, HDR, and then upgraded the Source Engine to the very next version, suddenly Half life 2 broke when it went from SOURCE 2004 to SOURCE 2005.
http://www.hlcomic.com/index.php?date=2005-10-05
Frame-rates dropped, things stoped working for some, stuttering came back. It was just a temporary glitch, but in essence it proved source failed in it's idealism that if you bought a game to work on a current gen system, you couldn't upgrade the games minium requirements past that current gen system.
Hell even the "streamlined" version of the engine from the xbox version was a problem for some.

i.e. the upgrades would in theory made HL2 stop working for those who purchased it in good faith.

Not to mention 3rd Party developers also had problems as they would branch off a certain build of the engine to develop their own engine. Sin Episode 1 is still on the 2004 Source engine where as HL2 is on the 2005 engine. Dark Messiah as well as Xeno clash sit on their own offshoots unable to be updated by the generic source upgrades I believe (oh and how dark messiah would greatly blossom with the ssbumpmapping and dynamic lighting of later builds)

And thus came the episodic builds into existence.
They would deliver the upgrades to the engine, but here is the rub,
Valve now would section off the builds for mod developers and third-party developers.
Whether this was the initial intention of the episodes to all sit on individual builds of the engine or rather a construct after the apparent failure of the HDR uprade in HL2, I can't say, It's purely conjecture from my part.


The result - the good, the bad and the ugly.

Even IF the episodic nature of the Half Life story has failed,
the yearly iteration of the Source engine towards it's long term goal of becomming valves, NEXT ENGINE, incidently called Source due to branding reasons of course, unless they want to break from that tradition, has been successfull.
Without EP1 valve could not have made TF2.
Without the orange box engine, valve could not have leapfrogged into Left 4 Dead.

These "tradionally" would have either have had to wait until 2010 for example, or be shoe-horned into the HL2 2004 original build of the engine, either way, neither are a pleasant prospect to consider.
So now we have circumvented any weird mod expirements like the counter-strike vehicles, as they get created into the engine-proper by valve.

Valve have kept up the technical side, first with the 64 bit build of the engine back in 2005, to the first multi-core version in 2007, to the hetrogenous version in 2008.

Whilst valve have ignored the graphical flourishes that most engines hype up, (such as crysis) to ensure things like physics,animation and AI work flawlessly (and questionably amongst some forum members) seeding the groundwork for believable characters in thier movemnts and logic, before making them look realistic.

This has confused some of the fan-base, believing the engine not to have taken it's innovative leaps that valve promised, because the innovation lies out of sight and aren't graphical in nature, whilst Third-party developers and well as modders curse the game breaking nature of the hops between yearly builds as well as the map compiler/lighting/rendering being a real hinderance to design (Ace-XENO clash whished valve would spend a bit more time on source's lighting and rendering, and Natural Selection 2 built their own engine citing they didn't have to wait 30-60 minutes for a map compile and everything was "dynamic").

This is somewhat a failure of source's idealistic concept of being state of the art, as visually valve have been lax with the art assets updates, and with the eye candy.
Don't get me wrong, it works and looks pretty in my honest opinion and it works for the masses, moving from a crappy old system to a new system, despite the eye candy, when I look back to my old system, I honestly don't feel like I missed out on much as I feared. (great job valve!)

The source engine now has the capacity to move physics calculations OFF the CPU and onto the GPU and move Rendering OFF the GPU and onto multiple CPU's of modern systems, be it 32 or 64 bit.

Ideally, not paying attention to things like physics or AI, Valve has turned your computer into a rendering farm.

And now Valve (in those portal2 pictures) look like they are going for glory with the rendering and eye-candy.

So does this mean that after 6 years source 1.0 has iterated towards the mythical 2.0 version? (based on the eye-candy)

Has this transition been a success? I feel it has.
I think valve's idealism hasn't worked out the way it wanted to (such as keeping HL2 state of the art) but the iterative and modular nature of the engine has kept valve yearly produce of new engine builds and games whilst keeping the technology up to date, has been an utter success.

senne teddy
03-13-2010, 03:36 PM
Ah yes, Boff is here again with some technical information about the source engine. And again very interesting.

wrage
03-13-2010, 05:01 PM
Excellently written. Nice job.

Whatthecell
03-13-2010, 05:19 PM
Very well written! I completely agree! Oh, and IGN has a similar article for those who are interested:
http://pc.ign.com/articles/102/1027317p1.html

LATTEH
03-13-2010, 06:53 PM
Honestly any game engine that is easy to mod for is an success in my book.


it makes it last way longer then any game that is developed by a company that has a Fantastic engine that looks beutiful but it is hard to mod for

Dagen
03-13-2010, 07:04 PM
Excellent.

Flamov
03-14-2010, 04:40 AM
Well done there. +rep

AguyinaRPG
03-14-2010, 07:53 AM
The Source engine is a triumph.

It looks great (hush nay-sayers), it keeps getting better, and it can run on even my piece of garbage comp (okay, I'm over-exaggerating). But the point is, we have the opportunity to have bigger gateways into better technologies because of the source engine.

Boff
03-14-2010, 11:56 AM
I had seen that article before, despite touching on it, they didn't really make the firm point of valve's strategy with the source engine, and then they eventually mentioned the "age" of the engine and that valve will go on updating and patching. For me it just gives the wrong impression (my own personal interpretation of course)

I really don't like the concept of patching and updating, especially in reference to source age, it just sounds like valve are streamlining an old piece of kit, you know like tightening up the nuts and bolts to make the existing stuff go that bit faster, when actually they are bit by bit building the so-called newer engine(naturally they ARE updating and patching the existing stuff to make it go faster all the same).

I hope valve at the next GDC do a post mortem on the source engine and discuss the aspirations, the idealism, (and not my own conjecture and supposition) the pitfalls, the criticism.

I would like for the third party developers to speak up,
and the mod community
to discus their views of the yearly progression of the engine.

for example:
I feel sorry for the city17 episode 1, credit should duly given to the people who hacked the orangebox torchlight to create realtime shadowing, only for valve to eventually (by the looks of it in source 2010) get around do it themselves.

Could a slightly better renderer have allievated or helped out small companies like ACE for xeno clash, or the natural selection people?

I naturally got a little grumpy with the call for a new engine all of a sudden by the influx of new people due to the portal 2 announcement. This is my rebuttle, an little information for those who weren't there at the begining, with half life 1 and half life 2 and saw how valve sidestepped the classics pitfalls and problems of game developments,
steam being one, and source being another, but where people give steam credit, source (due to a lack of eye candy and more specifically the lightmaps) gets ripped into for being out of date and valve not being smart with their engine, where as I think the opposite is implied.

dazza159
03-14-2010, 12:16 PM
If the renders of portal 2 are true to form then i think there should be a source 2010 display to show off the advancements that source 2010 brings to the table :P

meltra
03-14-2010, 12:58 PM
It feels like some days, there are only a handfull who champion the source engine, and with the new pictures of Portal2 floating around, it looks like valve have really put time and effort into eye candy and the rendering on the 2010 build of the engine.

But I consider Source, and I'm sure valve consider it, a success.

And this is a little For Your Information, for those who was not with, or just plainly forgotten the concept of Source.


Source : A Strategy and A concept

This is why I think source is a success, because you have to remember how and why the source engine EXISTS.

Valve created the source engine, to make HL2, but also to address and issue it had noticed that plagued the gaming industry.

Much in the same way they had problems with patching GldSRC engine for HL1/CS/TFC->TFC1.5 they envisioned STEAM, where they could distribute Patches, and act as DRM, and it not very popular from the begining, I think we can agree that steam as a strategy for valve, has been a success.

And thusly we come back to Source being a success, as it too was a strategy.

Not only did valve have to build and then patch for the old engine, but they had to develop the next generation engine and hit all the technical milemarks at the release.

And then after that, they had to build the 3rd engine, whilst patching engine number 2, for expansions and what not.

But valve opted NOT to do this traditional engine building.

SOURCE - a Brand name
Source got it's own funky logo, and it was going to take on multiple-personality syndromes with the following concepts, the brand-name would tie it all together so everything would be OKAY, or so valve very simarily thought about about the naming convention of the episodes......

CONCEPT 1a
What IF Valve created an engine, and ALL the games built on that engine would run on the same engine,

so a patch would only have be run out "once" to fix the engine via steam.

CONCEPT 2
IF the engine was built in a modular fashion, and theoretically, the modules would be replaced in time, so the OLD renderer get replaced with a more modern renderer that addresses the hardware at that future. The network code could be replaced, the sound system etc etc, until all the modules have been replaced, and suddenly you have a iterated into a NEW engine.
http://www.youtube.com/watch?v=bk24RdfXWcg

(oddly enough people still have problems with this concept, and still compare The source engine that powers HL2(2004/5) with games from 2009)

CONCEPT 1b Idealism - concept 1 revisited
Now that the all the source engine games used the same engine and would always be patched then thusly all the engine would always be upgraded, and of cousrse then all your source engine based games would be STATE OF THE ART since they ran of the same build of the engine.

You would also avoid the unpleasant scenario's of these shoe-horning in concepts into a engine that can not or barely can handle them.
If anyone remembers the APC's and other vehicle experiments in the original CounterStrike, as well as the map overlays and gui-systems that eventualy slipped into TFC 1.5.


SUCCESS AND FAILURE
As of 2005 concept 1b failed with The lost Coast.
As valve iterated toward the first major milestone of change, HDR, and then upgraded the Source Engine to the very next version, suddenly Half life 2 broke when it went from SOURCE 2004 to SOURCE 2005.
http://www.hlcomic.com/index.php?date=2005-10-05
Frame-rates dropped, things stoped working for some, stuttering came back. It was just a temporary glitch, but in essence it proved source failed in it's idealism that if you bought a game to work on a current gen system, you couldn't upgrade the games minium requirements past that current gen system.
Hell even the "streamlined" version of the engine from the xbox version was a problem for some.

i.e. the upgrades would in theory made HL2 stop working for those who purchased it in good faith.

Not to mention 3rd Party developers also had problems as they would branch off a certain build of the engine to develop their own engine. Sin Episode 1 is still on the 2004 Source engine where as HL2 is on the 2005 engine. Dark Messiah as well as Xeno clash sit on their own offshoots unable to be updated by the generic source upgrades I believe (oh and how dark messiah would greatly blossom with the ssbumpmapping and dynamic lighting of later builds)

And thus came the episodic builds into existence.
They would deliver the upgrades to the engine, but here is the rub,
Valve now would section off the builds for mod developers and third-party developers.
Whether this was the initial intention of the episodes to all sit on individual builds of the engine or rather a construct after the apparent failure of the HDR uprade in HL2, I can't say, It's purely conjecture from my part.


The result - the good, the bad and the ugly.

Even IF the episodic nature of the Half Life story has failed,
the yearly iteration of the Source engine towards it's long term goal of becomming valves, NEXT ENGINE, incidently called Source due to branding reasons of course, unless they want to break from that tradition, has been successfull.
Without EP1 valve could not have made TF2.
Without the orange box engine, valve could not have leapfrogged into Left 4 Dead.

These "tradionally" would have either have had to wait until 2010 for example, or be shoe-horned into the HL2 2004 original build of the engine, either way, neither are a pleasant prospect to consider.
So now we have circumvented any weird mod expirements like the counter-strike vehicles, as they get created into the engine-proper by valve.

Valve have kept up the technical side, first with the 64 bit build of the engine back in 2005, to the first multi-core version in 2007, to the hetrogenous version in 2008.

Whilst valve have ignored the graphical flourishes that most engines hype up, (such as crysis) to ensure things like physics,animation and AI work flawlessly (and questionably amongst some forum members) seeding the groundwork for believable characters in thier movemnts and logic, before making them look realistic.

This has confused some of the fan-base, believing the engine not to have taken it's innovative leaps that valve promised, because the innovation lies out of sight and aren't graphical in nature, whilst Third-party developers and well as modders curse the game breaking nature of the hops between yearly builds as well as the map compiler/lighting/rendering being a real hinderance to design (Ace-XENO clash whished valve would spend a bit more time on source's lighting and rendering, and Natural Selection 2 built their own engine citing they didn't have to wait 30-60 minutes for a map compile and everything was "dynamic").

This is somewhat a failure of source's idealistic concept of being state of the art, as visually valve have been lax with the art assets updates, and with the eye candy.
Don't get me wrong, it works and looks pretty in my honest opinion and it works for the masses, moving from a crappy old system to a new system, despite the eye candy, when I look back to my old system, I honestly don't feel like I missed out on much as I feared. (great job valve!)

The source engine now has the capacity to move physics calculations OFF the CPU and onto the GPU and move Rendering OFF the GPU and onto multiple CPU's of modern systems, be it 32 or 64 bit.

Ideally, not paying attention to things like physics or AI, Valve has turned your computer into a rendering farm.

And now Valve (in those portal2 pictures) look like they are going for glory with the rendering and eye-candy.

So does this mean that after 6 years source 1.0 has iterated towards the mythical 2.0 version? (based on the eye-candy)

Has this transition been a success? I feel it has.
I think valve's idealism hasn't worked out the way it wanted to (such as keeping HL2 state of the art) but the iterative and modular nature of the engine has kept valve yearly produce of new engine builds and games whilst keeping the technology up to date, has been an utter success.

I agree with the entire post. It is an interesting view of things.

Moose Muscles
03-16-2010, 08:23 AM
Well written. Good points. A+.

+rep

ben583
03-16-2010, 08:31 AM
Totally agree, good post.

Boff
07-20-2010, 01:26 AM
So with alien swarm turning up we got to see an aditional "glimpse" of the source(brand name) engine production

http://forums.steampowered.com/forums/showpost.php?p=16146455&postcount=40
Alien Swarm branched off of the main tree several months ago. A project will branch off in order to ensure stability so that no external changes outside of the project will impact it. For example, TF2, L4D, L4D2 all live in their own separate branches since they've released to ensure only updates to those specific projects go live and out to the public. It is mainly a quality and stability insurance.

So imagine if you will,

a large arrow strechting along a the longs side of a4 paper.
At the base of the arrow is the date 1996 ( i guess which is when they tookt the original quake engines and start to work on them)
At the bottom where the arrow is going is "THE FUTURE"


The arrow represents sources development.

Is it a straight line or a meandering river weaving around various milestones, hitting some, avoiding some tech (parallax shaders, vertex lighting) but aiming towards others.

First branch to come off this river 1998 Gold src

Then there was that mod destroying patch to the engine probably around late 1999 early 2000 ? second branch?

then you just tick off every game release
2004 Counter Strike Source
2004 Half life
2005 Lost Cost
2005 Day of Defeat
2006 Ep1
2007 TF2, EP2, Portal
2008 L4d1 (which doen't forget Dod:S gets flashlight and multicore as did TF2)
2009 L4d2
2010 well hl2 re-release 2009 build apparently and alien swarm which is a build between l4d2 and portal.


Obviously leading the charge is the R&D putting stuff into the valve film-maker before we see it put back into the game proper.
(Colour grain, depth of field, blurs, Tyndral (one day), dynamic lighting and shadowing, etc etc)

AoA
07-21-2010, 08:38 AM
so what your getting at here is that the episodes are valves biggest failure?

Boff
07-21-2010, 01:02 PM
actually I was talking about the yearly builds of the source engine.

cakewalker
07-22-2010, 11:21 AM
I agree with this thread.

A solid engine and concept for it, fitting the time we live in, is a win.

shresht93
07-22-2010, 11:46 AM
Hey, multicore rendering is still slower than single-core.

And they cancelled the 64-bit version for some reason.

Lewk
07-23-2010, 09:08 PM
And now Valve (in those portal2 pictures) look like they are going for glory with the rendering and eye-candy.

So does this mean that after 6 years source 1.0 has iterated towards the mythical 2.0 version? (based on the eye-candy)
Hi Boff! Lewk again....

Sorry to burst your bubble, but Portal 2 actually doesn't render much differently. The only difference you're seeing is the world shadows. They've replaced the current system where they bake the shadow into the bsp maps at compile time, and replaced it with a dynamic shadow map that now sits on top. The lighting is very largely unchanged, and the whole geometry (bsp) system is exactly the same.

It's different to they way they do the torch light shadow now, where that shadow is tied to the dynamic light.***

So now, instead of the world shadows being baked into the geometry and then the lighting compiled on top; the lighting is now compiled (the same way as it's always been done), and the shadow sits on top (in real time).

If valve were to "overhaul" the whole rendering, they would ditch the bsp method, scrapping hammer altogether. It's something you can see them working on, and something that is coming down the line, but not with Portal 2. I've written about that before...

-----

Anyway, the most intriguing part about all this is:

Now that the world shadow is applied in real-time on top of the lighting, AND all the Valve titles in the past 3 months were updated to the 2010 Source engine, you can effectively apply the dynamic shadow in all of Valves back catalog! Without having to re-compile all the maps. It'd just sit on top of the baked in shadows. All objects, npc's and particles will effectivle have real time dynamic shadows!!!!!!!

Now I cannot wait to play Half-Life 2 with dynamic shadows!




***With Alien Swarm and Portal 2, they're allowing you to use one of those (torch/flashlight) dynamic lights in each scene. If you play through Alien Swarm, you will see that they're spaced apart at enough space so they're only rendered on the screen at one time. This is to save on performance hits with lower end systems.

Boff
07-24-2010, 04:00 AM
(i know :D very excited) it was those lovely self shadow bump maps that they had it that tech paper :)
as you said They basically stated they could store shdaow data into the ssbumb instead of bake on the "shadows" as per standard old fashioned lightmap. However to be more effecient and effective as you rightly said. Use the map compile to generate the bulk of the information and then add that little finese in realtime. (they even said the "stored" info was more effective than baking but you needed shader model 3 onwards for it to work)

i was never sure that was actually the case as those tech papers are another language :)

but you actually see "switch" in the portal2 suction tube video. As the play walks into the last room, the shadowing switches from a simplified version (from the map compile) to the more realistic realtime version.

checking through alien swarm one still sees some classic "source-isms" such as sections of pipe being entirely lit when section of it fall into shadow (or vice-versa).
Very exciting times!!!

HyperTech
07-25-2010, 06:43 AM
and with the new pictures of Portal2 floating around, it looks like valve have really put time and effort into eye candy and the rendering on the 2010 build of the engine.
We will see, about that.

I remember the trailer for Portal 1, there was graphic effects in it which was removed in the retail version (or the effects might have been just pre-rendered, I'm not sure), to make people more excited about the game (just like how Valve faked the AI in the HL2 trailer back in 2003 to make people more excited). I bought the games just to see that the graphic effects (Portal 1) and the AI (Half-Life 2) are not there.

I do not believe or expect anything in Valve's trailers anymore, especially not in the graphic part. I will only cheer if I actually see the cool stuff in the retail version.