The Art of War, Online
Begin with strategy.
“No such thing as a
fair fight. Just fights you win and fights that kill you.”
Art of War Central came
into being by chance. We were one of the few teams with a dedicated server
machine to host our own game server. Of course, other teams soon wanted to
borrow our server to practice, scrimmage, and hold matches on. More and more
such requests came in. Soon, teams were even offering to pay to use the server.
The seed of AoWC was planted and growing. Over time, we developed a business
model that not only worked but flourished, growing as quickly as the PC online
multi-player game business grows. Game Server Hosting, by Art of War Central,
was born. Our name quickly spread through the community as the only people who
then provided such a service. We were struggling to equip ourselves with staff,
servers, bandwidth, and the many other things our research had shown us were
necessary. This was a journey into unknown lands for all of us. We had no
examples to follow, no standards to guide us, and the future of the game server
industry was at our fingertips (although we did not know it at the time). This
was originally supposed to be a hobby, and a way to recoup the cost of hosting
our own game server for our own use. It has since become its own creature, and
it known the taste of success.
Know the field of
battle.
“I got game.”
Let us begin with online multiplayer games.
Understanding what they are is crucial to understanding our business, and why we
are the best in it. A multi-player game—such as Counter-Strike, a
popular modification of Half-Life—is a game in which users from all
around the world play with each other over the internet or local area network.
They can play online together, their computers automatically sending data back
and forth to create the illusion of a seamless virtual world in which they all
coexist at once.
Some games do this by sending data from player
to player to player. This is called “peer to peer” networking. Most games,
however, use something called a dedicated server application. This dedicated
server app is a program specifically designed to collect data from each player
and redistribute it, as necessary, to the other players in the game. This is
much more efficient and effective than a peer-to-peer arrangement, but it
requires a separate computer to host the server application. And this new
computer is not a game machine. It is a server machine. It has very
special needs. To effectively host and support a game server, you must know
every aspect of its inner workings, on both the hardware and software sides of
the system.
Now, in the case of
Counter-Strike (as well as most other games), you can host a dedicated server
application on many different operating systems, and the primary bottleneck on
your server’s performance is bandwidth—how much data you can move into or out of
your computer at once. Most users that buy and play these games are using a
“broadband internet service,” the most popular of which are DSL and cable. With
a DSL or cable modem connection, a player can host his or her own dedicated
server application, but with this kind of a connection the server can only
“serve” a few people at a time, usually between four and ten. A server with
better internal hardware—a faster processor or more memory—might improve these
numbers a little, but the tightest bottleneck is still network bandwidth, and
cable and DSL have their limits.
In the past, this is how the majority of
game servers were hosted. This was the only option. The player would buy the
game, and most households only had one computer, so the player would use this
one machine host his or her server and play the game on, often
simultaneously. The stress on the computer was enormous, and game performance
was proportionately dismal. Even if the bandwidth on these newfangled broadband
internet services could keep up with the load, the computer itself was just
doing too much work internally. Chugging data for 3D graphics and physics and
sound and sorting and distributing network data to the other players—one
computer, especially back then, simply could not do it.
When these games become more popular,
especially as leagues, LAN events, and dedicated teams—with members scattered
all around the world—were created, the gamers started looking for something
better, a solution to the server dilemma. They needed something with the power
to get the numbers crunched and the data delivered, faster and more reliably.
They needed a solution that would give them true performance, with latency
return in tens of milliseconds (rather than hundreds or thousands) for all their
team members in games of up to twenty players. And more. Thirty. Forty. Sixty.
The player-count is still rising, even to this day, thanks to ambitious game
designers.
The solution was obvious. Use a professional
server. A computer designed to crunch numbers and transmit vast amounts of data
as fast as the users need it. And a few managed to do this. They managed to rent
dedicated servers to host their games. They paid between $200 and $700 a month
for the luxury, and the teams that could foot such bills were few and far
between, but the results were undeniable and wonderful. Silky-smooth gameplay.
Know the enemy.
“Bandwidth, bandwidth,
bandwidth.”
For that is the goal, the
golden fleece: silky smooth gameplay. A seamless union of computers to create a
seamless multiplayer world. The bur in that fleece is called “lag.”
Rushing down a hillside,
our gamer levels his weapon at an enemy some yards away, just then stepping from
an open doorway. The hapless foe turns as he steps out onto the doorsill, and he
sees our gamer, and he perceives that his doom is at hand. Our gamer’s gun-sight
centers above the target’s left eye. Everything goes still for an instant.
Make that a second.
Two seconds.
Five seconds later, our
gamer discovers that, while his screen was frozen (as his computer sat drumming
its virtual fingers, waiting for the data stream from the server to resume), he
has been shot by his would-be prey and left for dead on that fateful hillside.
That
is lag.
As we began to compose
our strategy to combat this enemy, we realized that more information was
required to host online games successfully. One cannot simply play a game and
expect to understand how to host it. A game server is, as stated earlier, a very
different beast from the game itself.
In truth, it is a simpler beast, a pared-down
version of the retail game software specifically designed to be a data
clearinghouse for the individual players. If you took an X-ray image of a game
server it would look very basic. Data comes in from an individual player’s
computer (called a client), and the server duplicates it and packages it with
data from other clients and then sends it all back out to the gamers. Thus, each
individual player’s computer receives the information it needs regarding the
other players—where they are and what they’re doing in the game world. If every
client gets the information it needs, the virtual world is seamless and smooth,
as if everyone were playing on a single computer.
Obviously, the individual client is not
sending much data to the server—only a few bits and bytes regarding that
player’s current disposition. An average DSL service will offer an outgoing
transfer rate 128kbps (kilobits per second), but
even at the height of a match, the client will only need to send perhaps 4kbps.
And at 256 kilobits to 1.5 megabits per second of incoming data, a client on a
DSL or cable service has plenty of bandwidth to receive all the data it needs
about the other players.
The servers themselves are not hard to build.
They have to be able to take data from as many as 60 (or more, nowadays)
separate clients, decide on the fly which clients need which data from which
other clients, duplicate and bundle data as necessary, and send these new
bundles out to their respective addresses. So, they have to be processing
workhorses. Simple. We give them huge processors (usually two huge
processors to a server) and huge memory. They also have to be able to send and
receive 60 or more data streams simultaneously, taking in and dishing out
relatively huge quantities of information. Again, simple. Give them massive
bandwidth by placing them in data-centers that are wired directly to “the
internet” via enormous optical pipes.
And yet our brave soldier still lies bleeding
on that hillside, another victim of lag. Where, then, does the lag come from? We
found our enemy in an unexpected but very logical place: the internet.
Take a look at this Traceroute.

It shows all the myriad places a single packet
has to go just to get from “here” to “there,” wherever here and there are. The
internet is a vast web, an intricate interweaving of connections from computer
to computer, and each packet of data has to find its way through that cobweb by
some miracle. Between two points on the internet, there is no straight line.
Therein lay our problem. Our brave soldier, as he sprints down the hillside, his
M4 carbine to his shoulder and his eyes fixed on the open door and the safe
hiding-place beyond it, is sending to the server a continuous stream of data
about his location, the direction he is facing, the speed at which he is moving,
what weapon he is holding, how many round remain in its magazine, etc. etc. And
his foe, stepping out of that doorway, is sending the same sort of information.
The server, recognizing that the two could interact with each other, is sorting
and resending each’s data to the other. Thus our hero’s computer has all the
necessary data to render that enemy on screen as he steps into view. Our hero
takes aim, and that change in his position is sent to the server, which sends it
to his enemy, who, caught off guard, can only watch in horror as his own
computer shows him this threat, bearing down on him from the hillside. Even as
he turns to ward off the attack, he knows he can never be fast enough.
And then, somewhere along the serpentine path
between the server and our hero, a string of packets runs into a traffic jam. By
some unfortunate turn of fate, this part of the data stream has been sent by a
roundabout route and finds itself in a bumper to bumper gridlock as, by another
turn of fate, it arrives at a router that is just then getting more through
traffic than it can handle. So this string of packets waits until the mess can
be sorted out and it can move on through and complete its journey, several
seconds later. Of course, by the time it arrives, the scene is over, and it that
little string of data is no longer relevant. In a game where frames per second
matter, several seconds of delay is an eternity. The obsolete data is ignored as
the client computer receives more recent information from the server. A whole
chunk of time is thus missing from the continuum, and our brave soldier does not
even witness his death. He only sees the world freeze for five long seconds,
before being cut to a view of his body and a countdown to respawn, while his foe
gives thanks for a miracle. From his perspective, certain death stops,
midstride, giving him time to level his own weapon and fire preemptively.
The machines were ready, on both ends. They
were capable. But somewhere in between, the system broke down.
Identify your targets.
“The shortest path
between two points may take you through a rock. Shoot the rock.”
Having identified the
enemy, we then set about figuring out how to combat it. That is where Internap
technology comes into play. We discovered that there were two primary causes of
lag. The first and foremost is “latency.” This is the actual time it takes for a
packet to make a round trip from client to server and back again.
A movie is just a series of still images. For
it to look like real life, rather than a slide-show, those images must be shown
in sequence at high speed, specifically twenty-four frames per second. The
average eye actually captures and sends to the brain between 40 and 50 images
per second, though, so the faster the better. 24 is good enough for cinema.
Television uses 30 frames per second. Modern videogames aim for 60 or higher,
and the best monitors can display a hundred or more.
But videogames do not have the luxury of
creating these images before-hand and then just reeling them off. They have to
create each frame in real-time, on the fly. Indeed, that’s the whole point. And
to do this, the game has to have enough information to draw each image. A
single-player game uses only information from within itself, within the
computer, so frame-rates in the hundreds are easily attainable. But a
multiplayer online game must wait for information from other computers before it
can draw a frame. If the time needed for data to travel from client to server to
client (again, called “latency”) is excessive, then each player will see and
excessive amount of time between images. The result of large latency times can
indeed be a slideshow. And one can not fight effectively when viewing the world
through a slideshow.
So latency must be kept
to a minimum.
The other chief cause of
lag is called “packet loss.” This term describes the actually loss of data in
transit. Either it arrives late and is ignored, or it disappears entirely, like
Rudolph Diesel crossing the English Channel. It gets on the boat but does not
get off at the other end, and is never heard from again.
Safe routes must be
found.
Choose your friends
wisely.
“Let no man stand
alone. After all, two guns are always better than one.”
Internap solves both of
these dilemmas for us, using advanced proprietary technology to locate the
shortest (fewest-hops) and safest (most-reliable-hops) path from the server to
each client. It then grabs and holds that path, unless a better one becomes
apparent, at which point it secures the new route and redirects the traffic.
Internap’s system is intelligent, dynamic, and robust. It is impossible to beat.
No one else has anything like it.
Develop tactics.
“Look, leap, land
safely, and laugh at those who did not.”
With all of this information and more at hand,
we set about creating a business plan that not only worked but would ensure the
survival and stability of our company. In the days of our genesis, we had to
rely on other people’s hardware. We would rent a dedicated server machine from
another company to host our game servers on, much as we now offer entire
dedicated servers for rent to our own clients. This was a good system at the
time, as the company was founded on very little capitol, and rental of servers
required no large initial investment and thus little risk as we sailed into
uncharted technological and strategic waters. At that time, there was no one
else out there trying what we were trying.
But we were paying a monthly fee for each
computer and the bandwidth it used, and this was a bad business model as we well
knew. If you rent, your costs never go down. We were paying and estimated $400
to $500 per dedicated server at the time, and as the hardware industry developed
(at a phenomenal rate), we continued to pay the same price for each of our
systems, even as the technology curve left them in its wake. The value of our
hardware decreased, but our monthly bills did not, and of course the bills never
stopped as long as we needed those machines.
However, once we found that there was indeed a
market for game servers (not just
dedicated server machines, but the hosting of the individual server applications
themselves), we changed our plan. We began to phase out the rented dedicated
servers and purchase our own hardware. Our costs began to decrease as we were
able to pay off each server investment within the first few months of use.
Select an arsenal.
“This is my server.
There are many like it, but this one is mine…”
In the past Art of War
Central has used several different specifications for servers. This has always
been one of our most important decisions, since these are the servers we give to
our customers, and these are the servers we must depend on to be reliable, fast,
and cost efficient. These are the servers on which we live. We have to be able
to tell our customers that we are giving them the highest performing servers
possible to host their game servers
on, and we have to mean every word of it. Over the past few years we have
changed much and stay on top to insure we are using the best possible hardware
specs.
To choose a server, one must consider how many
applications each server will run. The simple answer is “lots.” Dual processors
are a requirement. This is one of many reasons we have chosen to use the Intel
Xeon processor with Hyper-Threading technology. This new technology opens up two
processor threads for each processor. Previous processors could only complete
one task at a time (per processor). With Hyper-Threading technology our servers
are able to process four software threads at any given moment.
We use 2-3 Gigabytes of DDR Ram in each
server. This is very important, not only because the Windows 2003 Server
operating system uses over 300 megabytes of RAM all by itself, but also because
it gives us a safeguard against “thrashing.” Thrashing (also known as
“swapping,” or, in less technical lingo, “chugging”) is when a system is running
low on available physical memory and it begins to use its page file (a space on
the hard drive that acts as “virtual” memory) as system memory. Once the page
file is cycling it begins to swap commands intended for the processor from RAM
to hard drive then back to RAM. This not only uses more system memory (yes,
memory saving devices take up memory) but also requires hard-drive read and
write, which is egregiously slow compared to physical memory. 2-3 Gigabytes of
RAM is enough to run the operating system and several applications such as
game servers, voice servers, and FTP services for access to game files for
the users, all simultaneously, without the operating system having to resort to
the hard disk for virtual memory.
As mentioned above, we use Windows 2003
Server Edition for our operating system. Since the release of 2003 Server,
it has been proven more stable than Linux, and offers us a much larger range of
game servers we can host. This is
mainly because most games are themselves made for Windows operating systems, and
with the advances in 2003 over 2000, the Windows server platform has become the
most reliable and effective environment for our game servers.
Every miniscule detail is taken in to
consideration when building our servers. The hard drives with game server files
on them are kept separate from the hard drives containing the operating systems,
providing a failsafe compartmentalization for purposes of data recovery, backup,
and transfer. It also prevents the IDE/SCSI bus on which the game servers run
from being slowed down by the operating system usage.
Awareness is
everything.
“If you can’t see it,
you can’t shoot it, so keep your eyes open.”
To host over 600 game servers is a very large
task, and with this many game servers online, it is very difficult to monitor
each one. Instead of attempting this, we began to monitor all of our physical
servers. This is a much more scalable task, and one that is much more efficient.
Through our network monitoring system we scan all of our physical servers (in
real time) for the following:

·
CPU usage
·
Memory usage
·
Hard drive seek time
·
Current latency
·
Packet loss
·
Bandwidth usage
·
System up time
·
Hard drive(s) temperature
·
Motherboard temperature
·
CPU(s) temperature
Monitoring these items 24/7 gives us an
accurate gauge of the load we are placing on each individual server. This also
gives us an estimate of when we need to purchase new servers to prepare for more
customers. This is where we developed our network baseline. This baseline was
created to ensure our level of performance is kept at peek levels. If a system
exceeds any part of our network baseline we have a clear signal that action is
required to alleviate whatever part of the baseline was crossed. Our current
baseline minimum specs for our systems are as follows:
Server Baseline
·
Intel Xeon 2.4 GHZ dual processor
·
2 Gigs DDR PC-3200
·
80 Gig 7200 RPM hard drive
·
Windows 2003 server
·
100/1000 Network Interface Card (NIC)
Network Baseline
·
Cisco 2924 XL network switch
·
Cat. 6 network cable
Performance Baseline
·
Below 45% CPU usage
·
600 megabytes available system memory
·
6 gigabytes of available space on the operating system hard drive
·
40 gigabytes of space on the application (game server, voice server, file
server) hard drive.
These are our minimum requirements for our
servers and our network. As stated before, if any of these specifications are
not met the equipment is either upgraded or liquidated and a replacement
ordered.
With this working plan for our hardware, we
had to come up with the same plan for bandwidth. For the most part (in the past
and still today), dedicated server hosting companies bill by the gigabyte of
data transferred each month. By that system, Art of War Central would have to
pay for exactly how much data we transferred each month, which, as the demand
for game servers grew, was quickly becoming a very large figure.
We were doubling our numbers each month.
Although we were very excited to see our company doubling in size every month,
the costs were increasing to match. Another bold maneuver was needed, and we
came up with a plan to negotiate with the actual bandwidth providers, known as
“backbone providers.” We would turn ourselves into a hosting company of a kind
with Verio or Sprint. A bold move it was, too. We would have to create our own
networks from scratch and start opening more locations in different areas of the
country so our clients can get a good ping from any geographical location.
Over time we have used several different
backbone providers. The problem was that many networks were not built to host
games. Every hosting company was literally pushing us out of the door due to the
extreme amount of load we were placing on their hardware. We kept searching,
however, and eventually, we found our friend: Internap, with whom we are still
allied to this day.
provide the quality of service we promise, we
once again had to come up with something new.
Know your weaknesses
and compensate.
“You are not perfect.
But they don’t have to know that.”
Bandwidth is the majority of the cost. We have
to ensure we have high quality bandwidth to provide top performance. If you have
a team playing a scrimmage or in a league match you must be sure their game
server is performing well. Using Internap as our bandwidth backbone was the way
to go because of their unique and powerful routing technology.
Internap continually checks for the best route
from all the carrier options available, including the tier 1
backbones. Top 85% (i.e. virtually all) of Internet
destinations are probed every five minutes over each and
every one of the links and the actual best performing route
is chosen. The very best performance is guaranteed and our
pure bandwidth comes with a 100% uptime, no packet loss, and
under three milliseconds of latency from us to tier 1
backbones, SLA?. With other carriers you have to take extra
measures to manage BGP? traffic, and in most cases it is
impossible since, managing traffic is time consuming and
expensive. The carrier one chooses is very important since
every carrier has its own business model. Some carriers—like
Cogent for example—over subscribe their network routes by
200% or more. Obviously Cogent would not make a great
carrier for handling gamers. In our history we have used a
number of carriers for a range of products and in our
history using Internap has been the best solution, giving
our gamers the best possible network performance when
combined with our Cisco routers and switches.
Essentially, Internap bandwidth transit
will always have the very best performing route possible to the end destination
and is dynamically adjusted to route around any Internet backbone outages and
problems due to the consistent regular performance probing, all resulting in the
100% uptime/no packet loss/sub 3ms latency SLA?. Value transit does not include
the route probing and optimization on top of BGP?, and does not have an option
of going over some of the largest tier 1 backbone transit links.
Consequently, the right bandwidth carrier is the one most capable of handling
mission-critical business and enterprise applications and implementations that
require the lowest latency possible; this is why we have chosen to stick with
Internap as our backbone provider.
Control the flow of
battle.
“Dominate! This ain’t
no spectator sport.”
Once we had the hardware costs down, the right
network providers in many different locations, the demand for high performance
game servers and the marketing in place to pull in over 150 game server orders a
month, we created one of the most functional web pages on the internet to help
us close the sales and to help the customers control their game servers. In the
beginning no company provided a method to control their game server remotely.
This was a problem for us too, for a while. Finally our programming staff sat
down with the tech support, sales, and customers, to find out the needs and
wants of all. We came up with a plan to provide a n
engine that will allow everyone to do what they need to do to make it faster,
and easier.
Art of War Central announced SCMS
(Server Control Management System). This system was requested by clients to
offer:
·
Start/Stop/Restart their game servers
·
FTP straight from the interface
·
File management section so they can easily edit configuration settings on
the fly
Our tech support staff
requested a long list which at the time was not all applied, yet in the next
version our programming staff has assured us that all of this will work.
·
Automated setup of game servers, so once a client orders a game server,
the sales staff would just need to put in approval information and engine will
auto install and setup the customer’s server.
·
Improved support ticket system, the older version was a form customers
would fill out tech support would have to find all the information first then,
and didn’t keep records on hand so if customer was continuing to have same
problem tech support didn’t know. The new system on the other hand, client puts
in a support ticket, tech support gets it along with all customers information,
and can read past tickets.
·
Quick connect to the remote server. This would automatically connect to
remote desktop to the client’s server, tech support time on a single ticket
decreased average of 60 second on completing.
·
The ability for customers to have and option of reopening ticket if they
continue to have the same problem giving tech support a better understanding on
what’s going on, even if it’s a new tech looking into the issue.
Sales had requested
·
An online tracking system so they can see where each member was referred
from. As the website logs do a good job to find out where the traffic is coming
from, but even the biggest marketing deals, won’t actually provide customers.
This has given our sales staff to target the areas better on where the actual
sales come from. Our marketing staff has many advantages with this, as other
companies are looking for the most traffic we are looking for more customers.
Our sales team has lowered the cost on marketing expenses have also increased
sales, and sales leads with this feature.
·
Cancel request tracking system; During Art of War Centrals past we would
keep an average customer 3-4 months. For many different reasons, customers have
to leave at some point most with the same comment we will back, and they did 10%
of the time. The average reasons to leave are money problems, team broke up, or
trying another company. This information was hard to find out before the new
system as our sales staff didn’t have this information. It was in a database
though; sales staff demanded to know who is canceling why, and needed permission
to contact the clients. Once the system was implemented sales staff will check
cancellations daily, and have a chance to keep customers, and if not at least
have the ability to show how much we care. We have increased our average
customers staying to 5-6 months. Increased our recurring customers to 30%, most
clients will find a new team, start there own, or move to a new game.
·
The sales staff had also requested for there to be a chat system online
so when our staff is not busy they can actually talk to visitors that are on the
webpage. This feature is still beta and hard to do when you have so many people
talking to you at once. Currently, its not used much, but the system works like
a pop up ad, after the visitor clicks on at least 2-3 links on our site, this
guy may be interested but might leave because the client didn’t find what it was
looking for our, doesn’t know what he is looking for. The chat system will pop
up the customer ask the client if he would like to talk to a sales rep, they say
yes our staff is there at the other end to help the client, to find out what
they might be looking for, and since this is beta still we don’t have hard
facts, but each sales rep who has used it, has all agreed they helped close 2-3
sales on there shifts they used the chat system.
Many more features are in
the works, to help staff, customers, and partners.
Learn as much from
victory as defeat.
“A warrior is a
student of anatomy. So shoot with your eyes open and listen for the loudest
screams.”
Our customers pay for a game server that
provides them a place to play 24 hours a day 7 days a week with a low ping and
knowledgeable game administrator’s. They pay for the customer service they get
by contracting for player-capacity.
Hardware and bandwidth usage is thus paid for
by the client, month we use our profit to expand or upgrade hardware or to offer
free services to the community, in order to improve and expand our public image
& build the onling gaming community up and our position within the market.

Grow.
“He’s good. In fact,
he’s the best. And yet…he could be better.”
We do grow. Not as quickly when Art of War
Central was first formed, for at that time we were the only game server company
up and running. Yet we still grow, and stand highest among the few companies
that can support actual paid staff members, office, toll free numbers and many
other differences about our company that put us on top. Many other providers are
still trying to push the cost down for the gamer, and to do so they are pushing
their staff numbers down, and pushing their performance down. Art of War Central
is pushing performance, and customer experience, which is what the customers
really want, up. The client will look for the lowest cost game server, but they
tend to get tired of that after awhile, and we many of our customers from our
cheaper competitors. Indeed, we find that those who leave us in favor of a
cheaper solution come back within three months, their main reasons (by their
testimony) being our superior customer support and performance.
With Art of War Central you are paying
for a game server. Art of War Central worries about the hardware
cost, and the cost of hardware investment is leveled over the entire customer
base, preventing any one customer from getting…well…ripped off. With Art of
War Central, we do that work for you and your clients. Art of War Central
will make sure your servers are running at top notch performance and that you
are not overloading any hardware for your game servers. With Art of War
Central our staff is there 24/7 monitoring not only the hardware, but each
game server on that hardware.
With Art of War Central there is no
bandwidth cap. No limit
Art of War
Central offers the better solution to your game server
needs as a whole. Better by far. State-of-the-art, custom hardware, unlimited
bandwidth, 24/7 customer support—these are just the beginning. Art of War
Central’s game server hosting is second to none. Game Servers are our
business. We have been hosting game servers since 2000, and around them we have
built a business model that is flourishing. Our staff has the experience needed
and no priority other than to provide beautifully performing game servers and
the best support for those servers in existence.
High performance, low
cost, professional support. The Art of your War, perfected.
|