Mod backup server
matso
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
This is the README.txt for the proposed Steam backup server.
Notice that this supports server ops that wishes to offer a backup if steam acts
up. While it would of course be better to have a few official backup servers, the
CDT does not have access to hosting, so it will be up to server ops to find that.
It should be really painless to setup and maintain a server, and it should be easy
for groups of server ops to join up and cross-backup each others servers.
The aim is for the server (and the support in the NS2 server/client) to be shipped
in 273 which hopefully should be out in January.
----
======================
SteamModBackup for NS2
======================
The SteamModBackup server ensures that NS2 clients will be able to download
a server-side mod even when Steam is acting up OR when the server is running
an out-dated version of a mod.
Its intended use is to make it simple for anyone to host a steam backup server
for their own or others servers.
To support "swapping" backups between groups of server ops, multiple backup
servers can be listed - as clients accesses the servers in a random order, load
will be automatically distributed over listed servers.
The server requires nearly zero maintenance, as the set of mods backed up comes
from client requests and missing mods are automatically downloaded from steam.
As the server is written in plain python and just extends / restricts the
standard python webserver, it should be easy to verify that it is safe to run.
Installing
NS2:
In ServerConfig.json, set "mod_backup_servers" to the list of backup servers.
"mod_backup_servers":[ "http://example.com:27020", "http://example2.com:27020" ],
If you would like people connecting to your server to download directly from the backup
servers without bothering to try steam first (faster at all times, but especially so
during steam sales), you can change set the "mod_backup_before_steam" key to true, ie
"mod_backup_before_steam":true,
Backup server:
Requires python 3+ (https://www.python.org/downloads/)
Create a directory where you want the mods to be stored, place the
steammodbackup.py file there. By default the server will run on port
27020, edit the PORT variable in the file if you would like it to
be something else.
Run it, making sure it runs with the mod directory as its current
directory.
Also make sure it can be accessed through any firewalls - as it just uses
the standard HTTP protocol, only TCP needs to be open on the PORT.
Test
You should immediately see a log.txt and a log-err.txt appear in
the directory.
You can test that it works by opening a browser and typing
http://localhost:27020/m5f3f7c0_1348957670.zip
Should get a 202 error message, refresh a few times and it
should either return the zip file (or a 404 if the mod
has been updated).
The corresponding .zip file should exist in the directory.
Configuration
The server will only accept apps in the ALLOWED_APP_IDS list, by
default set to 4920 (NS2).
By default, the server will service all mods for NS2; if you would
like it to service only a limited set of mods, list them in the
ALLOWED_MOD_IDS list.
Maintenance
The server should require zero maintenance, apart from possibly
restarting the server every 6 month or so just to truncate the
log files.
Mods are downloaded as clients submit requests for them - no need to manually
update anything.
Old mods are cleared out about a week after the latest version comes out.
Expected resource usage
The server should be very resource-conservative. As it will only serve server-side mods,
and clients will only connect to the server if they don't have the mod already, the only
time it will see significant use is
- first time players at a server missing one or more server-side mods (aka steam sale time)
- when a server-side mod is updated.
Note that while most mods weighs in at the few-kb size, maps range in size from 5 to 50Mb.
==========
Background
==========
That Steam servers serving up mods gets overloaded whenever there are Steam
sales is an unfortunate fact of life. Even during normal days, attempts to
download mod information or mods will fail about one time in four. During sales,
that goes up to about nine out of ten.
The server checks for new versions of a mod on every map change. This means
that when a mod is updated, there will be a time when the version of a mod
that the server is using is simply not available on Steam.
How long the server will run with an outdated version may vary - while it
checks every map change, if steam is acting up it may be upto several hours
before the server actually manages to download the new version.
Solution
The server acts as a very restricted web server. It ONLY services requests
for files whose pattern is "/m{mod-id-in-hex}_{timestamp/version-in-decimal}.zip".
If a file matching that pattern is present, it is served via the usual http-protocol.
If the request does not match the pattern, it is 404'ed.
If the request DOES match the pattern, but there is no such file, a 202 (request accepted,
come back later) is returned and a request is sent to the steam-downloader part of
the server.
The steam downloader will (in the background) try to download information about the
given mod_id. Once steam answers, the response is analysed and stored away (persisted
in the content.json file), and if the mod should be served, downloaded and stored in
the current directory.
To avoid spamming steam too much, the server keeps track of invalid requests so multiple
requests to outdated, unserved or otherwise invalid mods are denied efficiently.
This means that the set of mods being backed up will automatically adjust to the set
of mods actually being used by servers backed up, without any need for manual updating.
Notice that this supports server ops that wishes to offer a backup if steam acts
up. While it would of course be better to have a few official backup servers, the
CDT does not have access to hosting, so it will be up to server ops to find that.
It should be really painless to setup and maintain a server, and it should be easy
for groups of server ops to join up and cross-backup each others servers.
The aim is for the server (and the support in the NS2 server/client) to be shipped
in 273 which hopefully should be out in January.
----
======================
SteamModBackup for NS2
======================
The SteamModBackup server ensures that NS2 clients will be able to download
a server-side mod even when Steam is acting up OR when the server is running
an out-dated version of a mod.
Its intended use is to make it simple for anyone to host a steam backup server
for their own or others servers.
To support "swapping" backups between groups of server ops, multiple backup
servers can be listed - as clients accesses the servers in a random order, load
will be automatically distributed over listed servers.
The server requires nearly zero maintenance, as the set of mods backed up comes
from client requests and missing mods are automatically downloaded from steam.
As the server is written in plain python and just extends / restricts the
standard python webserver, it should be easy to verify that it is safe to run.
Installing
NS2:
In ServerConfig.json, set "mod_backup_servers" to the list of backup servers.
"mod_backup_servers":[ "http://example.com:27020", "http://example2.com:27020" ],
If you would like people connecting to your server to download directly from the backup
servers without bothering to try steam first (faster at all times, but especially so
during steam sales), you can change set the "mod_backup_before_steam" key to true, ie
"mod_backup_before_steam":true,
Backup server:
Requires python 3+ (https://www.python.org/downloads/)
Create a directory where you want the mods to be stored, place the
steammodbackup.py file there. By default the server will run on port
27020, edit the PORT variable in the file if you would like it to
be something else.
Run it, making sure it runs with the mod directory as its current
directory.
Also make sure it can be accessed through any firewalls - as it just uses
the standard HTTP protocol, only TCP needs to be open on the PORT.
Test
You should immediately see a log.txt and a log-err.txt appear in
the directory.
You can test that it works by opening a browser and typing
http://localhost:27020/m5f3f7c0_1348957670.zip
Should get a 202 error message, refresh a few times and it
should either return the zip file (or a 404 if the mod
has been updated).
The corresponding .zip file should exist in the directory.
Configuration
The server will only accept apps in the ALLOWED_APP_IDS list, by
default set to 4920 (NS2).
By default, the server will service all mods for NS2; if you would
like it to service only a limited set of mods, list them in the
ALLOWED_MOD_IDS list.
Maintenance
The server should require zero maintenance, apart from possibly
restarting the server every 6 month or so just to truncate the
log files.
Mods are downloaded as clients submit requests for them - no need to manually
update anything.
Old mods are cleared out about a week after the latest version comes out.
Expected resource usage
The server should be very resource-conservative. As it will only serve server-side mods,
and clients will only connect to the server if they don't have the mod already, the only
time it will see significant use is
- first time players at a server missing one or more server-side mods (aka steam sale time)
- when a server-side mod is updated.
Note that while most mods weighs in at the few-kb size, maps range in size from 5 to 50Mb.
==========
Background
==========
That Steam servers serving up mods gets overloaded whenever there are Steam
sales is an unfortunate fact of life. Even during normal days, attempts to
download mod information or mods will fail about one time in four. During sales,
that goes up to about nine out of ten.
The server checks for new versions of a mod on every map change. This means
that when a mod is updated, there will be a time when the version of a mod
that the server is using is simply not available on Steam.
How long the server will run with an outdated version may vary - while it
checks every map change, if steam is acting up it may be upto several hours
before the server actually manages to download the new version.
Solution
The server acts as a very restricted web server. It ONLY services requests
for files whose pattern is "/m{mod-id-in-hex}_{timestamp/version-in-decimal}.zip".
If a file matching that pattern is present, it is served via the usual http-protocol.
If the request does not match the pattern, it is 404'ed.
If the request DOES match the pattern, but there is no such file, a 202 (request accepted,
come back later) is returned and a request is sent to the steam-downloader part of
the server.
The steam downloader will (in the background) try to download information about the
given mod_id. Once steam answers, the response is analysed and stored away (persisted
in the content.json file), and if the mod should be served, downloaded and stored in
the current directory.
To avoid spamming steam too much, the server keeps track of invalid requests so multiple
requests to outdated, unserved or otherwise invalid mods are denied efficiently.
This means that the set of mods being backed up will automatically adjust to the set
of mods actually being used by servers backed up, without any need for manual updating.
Comments
Possibly there could be a "preferred" mod IDs list, which would always be allowed, but other mods would be served as long as the servers (space/mod count) restrictions have not been hit.
Someone did not read the "Configuration" section :-)
In practice, as the server only services NS2 mods, and the largest NS2 mods weights in at 50Mb (maps) - it would be hard to overload a backup server even if you got it to download ALL the NS2 mods in the workshop.
Also note that you get about-ish 1 Terabyte hard-drive space per programmer hour these days...
I did considering adding an authentication layer with RSA keys and login and stuff... but realized that if someone wanted to screw over NS2, going for the backup servers rather than the servers themselves seems like a silly way of doing it.
So all I did was to make sure the server is really limited, that it can catch invalid requests quickly, and that it by default only serves up NS2 mods ...
I figure it should be good enough in practice.