Go Back   Steam Users' Forums > Steam Game Discussions > T - Z > Team Fortress 2 > Server Feedback

Reply
Click here to go to the first staff post in this thread.  
Thread Tools Display Modes
Old 05-10-2011, 01:54 PM   Click here to go to the next staff post in this thread.   #1
jonl
 
JonL
Join Date: Aug 2009
Reputation: 132
Posts: 86
Replay Overview for Server Operators & FAQ

Replay Overview for Server Operators

Intro

For a non-technical overview of the replay feature, you can view the press release here.

At its core, Replay is an extension of SourceTV. The feature must be enabled on the server side, and the "demo" files that are ultimately generated include the full PVS, meaning everyone's perspective is recorded. Regular demos, on the other hand, only record the local player's perspective.

At the beginning of the "setup" phase for a round of gameplay, the Replay system will begin recording if enabled (via the replay_enable ConVar). The round, also known as a recording session, is given a unique name based on the date and time of the recording. As the round/recording session progresses, small blocks of data are periodically published to a fileserver where they can be downloaded by clients.

Replay File Types

There are two types of files that are published:
1. Recording Session Blocks - These files are essentially compressed chunks of a demo file. When a client saves a Replay, the system figures out how many blocks it needs to download in order to play back the corresponding life. When the client goes to play back a particular Replay, the blocks are decompressed, validated, and written to one monolithic demo file for the game to play.
2. Session Info Files - Each round/recording session has one session info file, which maintain the state of all recording session blocks. The files use the extension .dmx, and are updated every time a block's status changes. Clients download the session info file for a particular replay multiple times if all blocks are not available (e.g. if they're still being published or not enough time has passed). Session info files also maintain a flag, indicating whether the session is currently recording or not.
There is no client-specific data in recording session blocks or session info files, since any resultant demo file contains everyone's perspective. Each client records its own player slot information, so that watching the replay can start from the correct first-person perspective.

When a client requests to save a replay, they must download all blocks, starting from the beginning of the round. For the curious, the reason for this is that the demo files are "delta compressed" - which means that every "frame" in the demo is dependent on the previous frame. While this is not ideal, it should be noted that clients are smart about which blocks need to be downloaded, meaning that for any particular round, each block will only be downloaded once.

Publishing

There are two ways a Replay server can publish files for clients to download:
1. Local Web server - This method requires the installation of a Web server on the same machine as the game server. To publish a given file, the server will write it to a temporary location on disk, and then move (rename) it to a local path that the Web server sees. If the rename operation fails, the system will explicitly copy to the destination and delete the source file.

The advantage of this approach is that some may find it more secure, and it requires less work to configure.

There are several disadvantages to consider here. The main issue is that clients downloading from the Web server may be using bandwidth and CPU needed to run the game server smoothly. This approach is also particularly costly for the game server in terms of bandwidth, since the cost is multiplied by however many clients are downloading the files. Lastly, this approach may open your server up to potential "Denial of Service" attacks if proper security measures are not taken. For faster machines with faster connections, this approach may be preferred.
2. Remote Web server/FTP Offloading - The alternative approach requires access to a second machine, which must be running both an FTP server and a Web server. The Replay system will asynchronously offload Replay files from the game server to this second machine via FTP. The Web server must point to the same location on disk so that it can serve the uploaded files to clients.

The main advantage to this approach is that the burden of serving the Replay data is lifted off of the game server machine and put onto another machine. The bandwidth cost is minimized, since each replay block file is only streamed (uploaded) once, vs the Local Web server approach, where, for example, every single client who saves a replay for a particular round is going to download the first block file.

FTP is not a secure protocol, so caution should be taken to make sure login credentials do not become vulnerable to exposure.
Configuring Replay

The HLDS Update Tool will install three .cfg files with detailed comments in each. You will only need to worry about two of the files.

replay_local_http.cfg - Edit this file if you plan on using the "Local Web server" method described above.
replay_ftp.cfg - Edit this file if you plan on using the "FTP Offloading" method described above.
replay.cfg - Edit the bottom of this file to execute either replay_local_http.cfg or replay_ftp.cfg. This is the main configuration file for Replay and should be executed on server launch (i.e. "+exec replay.cfg").

Regardless of which publishing method you choose for your server, you'll need to set the following three ConVars:
replay_fileserver_host
replay_fileserver_port
replay_fileserver_path

These ConVars are shared with clients and are used to construct the URL's for downloading recording session blocks and session info files.

The URL is constructed using the following as a template:
http://<replay_fileserver_host>:<replay_fileserver_port>< replay_fileserver_path>
So, as an example, if you have a Web server running on my.domain.com on port 80, you would set replay_fileserver_host to "my.domain.com" and replay_fileserver_port to 80. replay_fileserver_path should be set to the relative path where replay files will be published, so this would be something like "/" or "/replay". Note that none of these ConVars should contain "http://". You also do not want to include any slashes in your replay_fileserver_host.

If you cannot successfully point a Web browser to the URL constructed by replacing the template above with the values you have specified for the three fileserver ConVars, clients will not be able to download their Replays.

Configuring replay_local_http.cfg

You will only need to edit replay_local_fileserver_path, which is the full path for the Web server. For Apache on a Windows machine, using the replay_fileserver_* ConVars above, this path might be something like:
C:\MyWebServer\htdocs\replay
On Linux, this might be something like:
/path/to/my/web/server/htdocs/replay
Configuring replay_ftp.cfg

You will need to edit the following ConVars, which are used to login and upload and clean up Replay files:
replay_fileserver_offload_hostname - The host name of your FTP server.
replay_fileserver_offload_port - The port needed to connected to your FTP server. This is usually 21.
replay_fileserver_offload_login - Your FTP username.
replay_fileserver_offload_password - Your FTP password.
replay_fileserver_offload_remotepath - The path on the FTP server where Replay files should be uploaded. For example, "/users/myusername/replay"
Testing Your Replay Configuration

Once you've edited the config files, you can test everything by starting a dedicated server, making sure to include -replay to the command line parameters. This will execute replay.cfg automatically, and add 1 to the maximum number of allowed players. If you wish to execute a config file instead of replay.cfg, you may specify it as an argument to -replay. This is required if you are running more than one server off the same machine. For more info on this particular subject, see Running Multiple Servers on One Machine below.

Once the game launches, the Replay system will run a series of self-tests using your configuration. Depending on the publishing method you chose, the system will either try to publish a temporary file to your local HTTP path or publish a file to your FTP server.

A quick way to iterate on your configuration is to edit the pertinent replay*.cfg files and execute the replay.cfg file. The publish test will run any time replay_enable is set to 1, which is one thing that happens in replay.cfg.

Note that if a publish test resulting from setting replay_enable to 1 fails, replay_enable will be automatically set to 0. The only way to actually enable replay is by passing the publish test. This is to keep misconfigured servers from listing in the server browser as replay-enabled.

You can also force a publish test by executing the replay_dopublishtest command. This will run the test without affecting replay_enable.

Running Multiple Servers On One Machine

If this applies to you, be sure to take the following details into account:
  • You will need a different config file for each server.
  • If you are serving Replay data out of a local Web server, you will want to create a separate path for each server. You will need to point each corresponding config file to the appropriate path via replay_local_fileserver_path.
  • If you are serving Replay data out of a remote Web server (i.e. FTP offloading), you will want to create a directory on your FTP server for each game server. You will need to point each corresponding config file to the appropriate path via replay_fileserver_offload_remotepath.
  • You will need to start each instance of the server with a different -replay parameter.
As an example, say you have three servers running on the same machine.

Assuming you are running a local Web server out of /var/www, you will would want to create three dirs - for example, /var/www/1, /var/www/2, /var/www/3. You might have three config files: replay1.cfg, replay2.cfg and replay3.cfg.

For simplicity, you might put everything in a single config per server, rather than exec'ing other config files. Here's an example of what replay1.cfg might look like:
replay_enable "1"
replay_block_dump_interval "15"
replay_fileserver_protocol "http"

replay_fileserver_offload_enable "0"
replay_local_fileserver_path "/var/www/1"

replay_fileserver_host "myserver.com"
replay_fileserver_port "80"
replay_fileserver_path "/1"
You would want to start each instance of your server with a different -replayserverdir parameter. For example, your command-line arguments might be:
-replayserverdir server_1 -replay replay1.cfg
-replayserverdir server_2 -replay replay2.cfg
-replayserverdir server_3 -replay replay3.cfg
If you are running a remote Web server/offloading via FTP, you will want to create a new directory on the FTP server for each game server instance. For example, would like to upload replay data to your FTP server in the directory /home/misspauling, you might create /home/misspauling/1, /home/misspauling/2, /home/misspauling/3. You would then want to setup replay_fileserver_offload_remotepath in each config file. Your replay1.cfg file might look like this:
replay_enable "1"
replay_block_dump_interval "15"
replay_fileserver_protocol "http"

replay_fileserver_offload_enable "1"
replay_fileserver_offload_protocol "ftp"
replay_fileserver_offload_hostname "ftp.myserver.com"
replay_fileserver_offload_port "21"
replay_fileserver_offload_remotepath "/1"
replay_fileserver_offload_login "misspauling"
replay_fileserver_offload_password "secret"
replay_fileserver_offload_maxuploads "16"

replay_fileserver_host "myserver.com"
replay_fileserver_port "80"
replay_fileserver_path "/1"
Cleanup of Stale Replay Data

Replay files are considered "stale" if they were published more than replay_data_lifespan days ago. An asynchronous cleanup operation is launched in between every round, although it can be cleaned up explicitly using the replay_docleanup command.

FAQ

I want to run a Replay-enabled server. Where do I start? Read this document, from beginning to the end.

I'm running multiple servers on the same machine. Is there a way to specify a different directory for each server? Yes. See the section above for details (Running Multiple Servers On One Machine).

When a client presses the F6 key, what exactly does that do? On the client side, number of kills, spawn and death ticks, dominations, etc are maintained on a per-life basis. If a life goes by and the client does not press F6, the data is thrown away. If a client does press F6, however, this data gets flagged and is saved to disk. Once the player dies (if they weren't already dead when the pressed F6), the death tick is written and the system figures out the maximum recording session block needed to reconstruct a demo that will cover the entire life.

If no one saves a Replay for a particular round, does the data get thrown away? Yes.

Does Replay record on an empty server? No. Replay begins recording at the start of the "setup" phase and ends with each round.

If clients disconnect from my server, will they still be able to download their Replay data? Yes. Since Replay data is served on a Web server (and not on the game server itself), clients can download replays after disconnecting.

Will executing a changelevel command disrupt recording or publishing of Replay data? No.

Will shutting down my server mess up Replays? If you shut down your server using the "exit" command, it should publish any data it needs to and shut down gracefully. The "quit" command should work fine, too.

Is there a way to access the format for the recording session block files and/or session info files? See below.

Is it OK to kick the replay bot? Kicking the replay bot will break Replay and is unsupported.

I'm having issues port collisions. What's happening? You might try running the server with -replay_port 9000 or whatever port number you wish. Replay takes up a port which may collide with other games.

If you have additional questions, you can mail the HLDS list (sign up here) or post to the forums. If you don't get answers there, send mail to jonl@valvesoftware.com.
jonl is offline  
Reply With Quote
Old 05-10-2011, 02:37 PM   #2
Slyguy42o
 
Join Date: Jul 2003
Reputation: 4
Posts: 491
Does the replay feature use a player slot when in use?
Slyguy42o is offline   Reply With Quote
Old 05-10-2011, 02:38 PM   #3
socket
 
Join Date: Jan 2009
Reputation: 0
Posts: 23
Excellent reference material. thanks.

I have a question.
You mentioned:

Quote:
Once you've edited the config files, you can test everything by start a dedicated server, making sure to include +exec replay.cfg to the command line parameters.
Would there be any issues using in a server.cfg ?
Adding "exec relay.cfg" to it.

This way i would think if they comment out "exec relay.cfg" in the server.cfg the relay system would not be loaded on a map change.

Of course the replay_enable would need to be set to "0" by default.

Also what about this:

Quote:
Will shutting down my server mess up Replays? If you shut down your server using the "exit" command, it should publish any data it needs to and shut down gracefully.

Does that include using the command "quit" ?
socket is offline   Reply With Quote
Old 05-10-2011, 02:59 PM   #4
Slyguy42o
 
Join Date: Jul 2003
Reputation: 4
Posts: 491
another question, I have a server in which the rounds never end and the map never changes. Is this going to present a problem as replay never gets a chance to write files at round end?
Slyguy42o is offline   Reply With Quote
Old 05-10-2011, 03:49 PM   #5
Powerlord
 
 
 
Join Date: Jun 2008
Reputation: 16
Posts: 180
Quote:
Originally Posted by socket View Post
Excellent reference material. thanks.

I have a question.
You mentioned:



Would there be any issues using in a server.cfg ?
Adding "exec relay.cfg" to it.

This way i would think if they comment out "exec relay.cfg" in the server.cfg the relay system would not be loaded on a map change.

Of course the replay_enable would need to be set to "0" by default.
autoexec.cfg would be a better place to call exec replay.cfg from.
Powerlord is offline   Reply With Quote
Old 05-10-2011, 04:04 PM   #6
Wolfmark
 
 
 
Join Date: Jan 2008
Reputation: 8
Posts: 20
1. Is there any protection against "never ending" rounds?

You said that the first block is a start point and all following blocks are "delta compressed", but is there a reset or something like this after a while? This is important because if the start point is created only when a round starts then a player must download all the block files created during a round when requests the replay for the last life from that round.

2. For maps with multiple stages, a "round" is a full round or mini-round?

3. The mechanism has protection against partial/invalid files? For example the session info files are updated periodically (probably whenever a new block file is created), but theoretically a client may request the file from web server before the FTP upload ends...

Last edited by Wolfmark: 05-10-2011 at 04:07 PM.
Wolfmark is offline   Reply With Quote
Old 05-10-2011, 04:23 PM   Click here to go to the next staff post in this thread.   #7
jonl
 
JonL
Join Date: Aug 2009
Reputation: 132
Posts: 86
Quote:
Originally Posted by socket View Post
Excellent reference material. thanks.

I have a question.
You mentioned:



Would there be any issues using in a server.cfg ?
Adding "exec relay.cfg" to it.

This way i would think if they comment out "exec relay.cfg" in the server.cfg the relay system would not be loaded on a map change.

Of course the replay_enable would need to be set to "0" by default.

Also what about this:




Does that include using the command "quit" ?
You might be able to put
replay_enable 0
//exec replay.cfg
in your server.cfg. I haven't tested this, so I'm unsure if it will work. I don't see why it wouldn't, though.

The quit command is fine. I'll update the post. Good catch.
jonl is offline   Reply With Quote
Old 05-10-2011, 04:37 PM   Click here to go to the next staff post in this thread.   #8
jonl
 
JonL
Join Date: Aug 2009
Reputation: 132
Posts: 86
Quote:
Originally Posted by Slyguy42o View Post
another question, I have a server in which the rounds never end and the map never changes. Is this going to present a problem as replay never gets a chance to write files at round end?
Clients should be able to download their replays without any trouble, but since the round never ends, you'll miss out on the cleanup phase. For that, you can use replay_docleanup.

Also, this is generally problematic because clients could be downloading for a very long time.
jonl is offline   Reply With Quote
Old 05-10-2011, 04:44 PM   #9
QWCA
 
Join Date: Jul 2010
Reputation: 460
Posts: 2,042
Hey jonl, Can we use these takes/demos that we save as replays OUTSIDE of the normal editor so that we can setup advanced paths for cameras to follow and still benefit from the advantages of every perspective being recorded? Or is this camera pathing a planned feature for the default replay editor?
QWCA is offline   Reply With Quote
Old 05-10-2011, 04:50 PM   #10
mes_gots_its
 
 
 
Join Date: Mar 2010
Reputation: 317
Posts: 645
Very nice, hopefully this will enable more servers to be able to allow Replays to be recorded.
mes_gots_its is offline   Reply With Quote
Old 05-10-2011, 05:07 PM   Click here to go to the next staff post in this thread.   #11
jonl
 
JonL
Join Date: Aug 2009
Reputation: 132
Posts: 86
Quote:
Originally Posted by QWCA View Post
Hey jonl, Can we use these takes/demos that we save as replays OUTSIDE of the normal editor so that we can setup advanced paths for cameras to follow and still benefit from the advantages of every perspective being recorded? Or is this camera pathing a planned feature for the default replay editor?
This isn't supported but should work fine.
jonl is offline   Reply With Quote
Old 05-10-2011, 05:13 PM   Click here to go to the next staff post in this thread.   #12
jonl
 
JonL
Join Date: Aug 2009
Reputation: 132
Posts: 86
3. The mechanism has protection against partial/invalid files? For example the session info files are updated periodically (probably whenever a new block file is created), but theoretically a client may request the file from web server before the FTP upload ends...[/QUOTE]

Clients download the session info file every minute or so.

Clients wait until the session info file says that a given recording session block is ready for download.
jonl is offline   Reply With Quote
Old 05-10-2011, 05:24 PM   #13
QWCA
 
Join Date: Jul 2010
Reputation: 460
Posts: 2,042
Quote:
Originally Posted by jonl View Post
This isn't supported but should work fine.
I tried opening up a replay from the replay folder in the demoui but just get stuck in a map camera.
QWCA is offline   Reply With Quote
Old 05-10-2011, 05:47 PM   #14
Badonga
 
Guest
Posts: n/a
jonl, you or the team have to do something about the player slot. I mean servers that are FULL with players will be displayed as 1 slot empty so basically now you can't do the good old leave the PC and select the Automatically connects to the server when a slot is empty option.
  Reply With Quote
Old 05-10-2011, 06:08 PM   Click here to go to the next staff post in this thread.   #15
jonl
 
JonL
Join Date: Aug 2009
Reputation: 132
Posts: 86
Quote:
Originally Posted by Badonga View Post
jonl, you or the team have to do something about the player slot. I mean servers that are FULL with players will be displayed as 1 slot empty so basically now you can't do the good old leave the PC and select the Automatically connects to the server when a slot is empty option.
We are working on the issue.
jonl is offline   Reply With Quote
Reply

Go Back   Steam Users' Forums > Steam Game Discussions > T - Z > Team Fortress 2 > Server Feedback


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



All times are GMT -7. The time now is 07:15 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
Site Content Copyright Valve Corporation 1998-2014, All Rights Reserved.