The Art of War, Online

Begin with strategy.

“No such thing as a fair fight. Just fights you win and fights that kill you.”

Art of War Central came into being by chance. We were one of the few teams with a dedicated server machine to host our own game server. Of course, other teams soon wanted to borrow our server to practice, scrimmage, and hold matches on. More and more such requests came in. Soon, teams were even offering to pay to use the server. The seed of AoWC was planted and growing. Over time, we developed a business model that not only worked but flourished, growing as quickly as the PC online multi-player game business grows. Game Server Hosting, by Art of War Central, was born. Our name quickly spread through the community as the only people who then provided such a service. We were struggling to equip ourselves with staff, servers, bandwidth, and the many other things our research had shown us were necessary. This was a journey into unknown lands for all of us. We had no examples to follow, no standards to guide us, and the future of the game server industry was at our fingertips (although we did not know it at the time). This was originally supposed to be a hobby, and a way to recoup the cost of hosting our own game server for our own use. It has since become its own creature, and it known the taste of success.

Know the field of battle.

“I got game.”

Let us begin with online multiplayer games. Understanding what they are is crucial to understanding our business, and why we are the best in it. A multi-player game—such as Counter-Strike­, a popular modification of Half-Life—is a game in which users from all around the world play with each other over the internet or local area network. They can play online together, their computers automatically sending data back and forth to create the illusion of a seamless virtual world in which they all coexist at once.

Some games do this by sending data from player to player to player. This is called “peer to peer” networking. Most games, however, use something called a dedicated server application. This dedicated server app is a program specifically designed to collect data from each player and redistribute it, as necessary, to the other players in the game. This is much more efficient and effective than a peer-to-peer arrangement, but it requires a separate computer to host the server application. And this new computer is not a game machine. It is a server machine. It has very special needs. To effectively host and support a game server, you must know every aspect of its inner workings, on both the hardware and software sides of the system.

Now, in the case of Counter-Strike (as well as most other games), you can host a dedicated server application on many different operating systems, and the primary bottleneck on your server’s performance is bandwidth—how much data you can move into or out of your computer at once. Most users that buy and play these games are using a “broadband internet service,” the most popular of which are DSL and cable. With a DSL or cable modem connection, a player can host his or her own dedicated server application, but with this kind of a connection the server can only “serve” a few people at a time, usually between four and ten. A server with better internal hardware—a faster processor or more memory—might improve these numbers a little, but the tightest bottleneck is still network bandwidth, and cable and DSL have their limits.

In the past, this is how the majority of game servers were hosted. This was the only option. The player would buy the game, and most households only had one computer, so the player would use this one machine host his or her server and play the game on, often simultaneously. The stress on the computer was enormous, and game performance was proportionately dismal. Even if the bandwidth on these newfangled broadband internet services could keep up with the load, the computer itself was just doing too much work internally. Chugging data for 3D graphics and physics and sound and sorting and distributing network data to the other players—one computer, especially back then, simply could not do it.

When these games become more popular, especially as leagues, LAN events, and dedicated teams—with members scattered all around the world—were created, the gamers started looking for something better, a solution to the server dilemma. They needed something with the power to get the numbers crunched and the data delivered, faster and more reliably. They needed a solution that would give them true performance, with latency return in tens of milliseconds (rather than hundreds or thousands) for all their team members in games of up to twenty players. And more. Thirty. Forty. Sixty. The player-count is still rising, even to this day, thanks to ambitious game designers.

The solution was obvious. Use a professional server. A computer designed to crunch numbers and transmit vast amounts of data as fast as the users need it. And a few managed to do this. They managed to rent dedicated servers to host their games. They paid between $200 and $700 a month for the luxury, and the teams that could foot such bills were few and far between, but the results were undeniable and wonderful. Silky-smooth gameplay.

Know the enemy.

“Bandwidth, bandwidth, bandwidth.”

For that is the goal, the golden fleece: silky smooth gameplay. A seamless union of computers to create a seamless multiplayer world. The bur in that fleece is called “lag.”

Rushing down a hillside, our gamer levels his weapon at an enemy some yards away, just then stepping from an open doorway. The hapless foe turns as he steps out onto the doorsill, and he sees our gamer, and he perceives that his doom is at hand. Our gamer’s gun-sight centers above the target’s left eye. Everything goes still for an instant.

Make that a second.

Two seconds.

Five seconds later, our gamer discovers that, while his screen was frozen (as his computer sat drumming its virtual fingers, waiting for the data stream from the server to resume), he has been shot by his would-be prey and left for dead on that fateful hillside.

That is lag.

As we began to compose our strategy to combat this enemy, we realized that more information was required to host online games successfully. One cannot simply play a game and expect to understand how to host it. A game server is, as stated earlier, a very different beast from the game itself.

In truth, it is a simpler beast, a pared-down version of the retail game software specifically designed to be a data clearinghouse for the individual players. If you took an X-ray image of a game server it would look very basic. Data comes in from an individual player’s computer (called a client), and the server duplicates it and packages it with data from other clients and then sends it all back out to the gamers. Thus, each individual player’s computer receives the information it needs regarding the other players—where they are and what they’re doing in the game world. If every client gets the information it needs, the virtual world is seamless and smooth, as if everyone were playing on a single computer.

Obviously, the individual client is not sending much data to the server—only a few bits and bytes regarding that player’s current disposition. An average DSL service will offer an outgoing transfer rate 128kbps (kilobits per second), but even at the height of a match, the client will only need to send perhaps 4kbps. And at 256 kilobits to 1.5 megabits per second of incoming data, a client on a DSL or cable service has plenty of bandwidth to receive all the data it needs about the other players.

The servers themselves are not hard to build. They have to be able to take data from as many as 60 (or more, nowadays) separate clients, decide on the fly which clients need which data from which other clients, duplicate and bundle data as necessary, and send these new bundles out to their respective addresses. So, they have to be processing workhorses. Simple. We give them huge processors (usually two huge processors to a server) and huge memory. They also have to be able to send and receive 60 or more data streams simultaneously, taking in and dishing out relatively huge quantities of information. Again, simple. Give them massive bandwidth by placing them in data-centers that are wired directly to “the internet” via enormous optical pipes.

And yet our brave soldier still lies bleeding on that hillside, another victim of lag. Where, then, does the lag come from? We found our enemy in an unexpected but very logical place: the internet.

Take a look at this Traceroute.

It shows all the myriad places a single packet has to go just to get from “here” to “there,” wherever here and there are. The internet is a vast web, an intricate interweaving of connections from computer to computer, and each packet of data has to find its way through that cobweb by some miracle. Between two points on the internet, there is no straight line. Therein lay our problem. Our brave soldier, as he sprints down the hillside, his M4 carbine to his shoulder and his eyes fixed on the open door and the safe hiding-place beyond it, is sending to the server a continuous stream of data about his location, the direction he is facing, the speed at which he is moving, what weapon he is holding, how many round remain in its magazine, etc. etc. And his foe, stepping out of that doorway, is sending the same sort of information. The server, recognizing that the two could interact with each other, is sorting and resending each’s data to the other. Thus our hero’s computer has all the necessary data to render that enemy on screen as he steps into view. Our hero takes aim, and that change in his position is sent to the server, which sends it to his enemy, who, caught off guard, can only watch in horror as his own computer shows him this threat, bearing down on him from the hillside. Even as he turns to ward off the attack, he knows he can never be fast enough.

And then, somewhere along the serpentine path between the server and our hero, a string of packets runs into a traffic jam. By some unfortunate turn of fate, this part of the data stream has been sent by a roundabout route and finds itself in a bumper to bumper gridlock as, by another turn of fate, it arrives at a router that is just then getting more through traffic than it can handle. So this string of packets waits until the mess can be sorted out and it can move on through and complete its journey, several seconds later. Of course, by the time it arrives, the scene is over, and it that little string of data is no longer relevant. In a game where frames per second matter, several seconds of delay is an eternity. The obsolete data is ignored as the client computer receives more recent information from the server. A whole chunk of time is thus missing from the continuum, and our brave soldier does not even witness his death. He only sees the world freeze for five long seconds, before being cut to a view of his body and a countdown to respawn, while his foe gives thanks for a miracle. From his perspective, certain death stops, midstride, giving him time to level his own weapon and fire preemptively.

The machines were ready, on both ends. They were capable. But somewhere in between, the system broke down.

Identify your targets.

“The shortest path between two points may take you through a rock. Shoot the rock.”

Having identified the enemy, we then set about figuring out how to combat it. That is where Internap technology comes into play. We discovered that there were two primary causes of lag. The first and foremost is “latency.” This is the actual time it takes for a packet to make a round trip from client to server and back again.

A movie is just a series of still images. For it to look like real life, rather than a slide-show, those images must be shown in sequence at high speed, specifically twenty-four frames per second. The average eye actually captures and sends to the brain between 40 and 50 images per second, though, so the faster the better. 24 is good enough for cinema. Television uses 30 frames per second. Modern videogames aim for 60 or higher, and the best monitors can display a hundred or more.

But videogames do not have the luxury of creating these images before-hand and then just reeling them off. They have to create each frame in real-time, on the fly. Indeed, that’s the whole point. And to do this, the game has to have enough information to draw each image. A single-player game uses only information from within itself, within the computer, so frame-rates in the hundreds are easily attainable. But a multiplayer online game must wait for information from other computers before it can draw a frame. If the time needed for data to travel from client to server to client (again, called “latency”) is excessive, then each player will see and excessive amount of time between images. The result of large latency times can indeed be a slideshow. And one can not fight effectively when viewing the world through a slideshow.

So latency must be kept to a minimum.

The other chief cause of lag is called “packet loss.” This term describes the actually loss of data in transit. Either it arrives late and is ignored, or it disappears entirely, like Rudolph Diesel crossing the English Channel. It gets on the boat but does not get off at the other end, and is never heard from again.

Safe routes must be found.

Choose your friends wisely.

“Let no man stand alone. After all, two guns are always better than one.”

Internap solves both of these dilemmas for us, using advanced proprietary technology to locate the shortest (fewest-hops) and safest (most-reliable-hops) path from the server to each client. It then grabs and holds that path, unless a better one becomes apparent, at which point it secures the new route and redirects the traffic. Internap’s system is intelligent, dynamic, and robust. It is impossible to beat. No one else has anything like it.

Develop tactics.

“Look, leap, land safely, and laugh at those who did not.”

With all of this information and more at hand, we set about creating a business plan that not only worked but would ensure the survival and stability of our company. In the days of our genesis, we had to rely on other people’s hardware. We would rent a dedicated server machine from another company to host our game servers on, much as we now offer entire dedicated servers for rent to our own clients. This was a good system at the time, as the company was founded on very little capitol, and rental of servers required no large initial investment and thus little risk as we sailed into uncharted technological and strategic waters. At that time, there was no one else out there trying what we were trying.

But we were paying a monthly fee for each computer and the bandwidth it used, and this was a bad business model as we well knew. If you rent, your costs never go down. We were paying and estimated $400 to $500 per dedicated server at the time, and as the hardware industry developed (at a phenomenal rate), we continued to pay the same price for each of our systems, even as the technology curve left them in its wake. The value of our hardware decreased, but our monthly bills did not, and of course the bills never stopped as long as we needed those machines.

However, once we found that there was indeed a market for game servers (not just dedicated server machines, but the hosting of the individual server applications themselves), we changed our plan. We began to phase out the rented dedicated servers and purchase our own hardware. Our costs began to decrease as we were able to pay off each server investment within the first few months of use.

Select an arsenal.

“This is my server. There are many like it, but this one is mine…”

In the past Art of War Central has used several different specifications for servers. This has always been one of our most important decisions, since these are the servers we give to our customers, and these are the servers we must depend on to be reliable, fast, and cost efficient. These are the servers on which we live. We have to be able to tell our customers that we are giving them the highest performing servers possible to host their game servers on, and we have to mean every word of it. Over the past few years we have changed much and stay on top to insure we are using the best possible hardware specs.

To choose a server, one must consider how many applications each server will run. The simple answer is “lots.” Dual processors are a requirement. This is one of many reasons we have chosen to use the Intel Xeon processor with Hyper-Threading technology. This new technology opens up two processor threads for each processor. Previous processors could only complete one task at a time (per processor). With Hyper-Threading technology our servers are able to process four software threads at any given moment.

We use 2-3 Gigabytes of DDR Ram in each server. This is very important, not only because the Windows 2003 Server operating system uses over 300 megabytes of RAM all by itself, but also because it gives us a safeguard against “thrashing.” Thrashing (also known as “swapping,” or, in less technical lingo, “chugging”) is when a system is running low on available physical memory and it begins to use its page file (a space on the hard drive that acts as “virtual” memory) as system memory. Once the page file is cycling it begins to swap commands intended for the processor from RAM to hard drive then back to RAM. This not only uses more system memory (yes, memory saving devices take up memory) but also requires hard-drive read and write, which is egregiously slow compared to physical memory. 2-3 Gigabytes of RAM is enough to run the operating system and several applications such as game servers, voice servers, and FTP services for access to game files for the users, all simultaneously, without the operating system having to resort to the hard disk for virtual memory.

As mentioned above, we use Windows 2003 Server Edition for our operating system. Since the release of 2003 Server, it has been proven more stable than Linux, and offers us a much larger range of game servers we can host. This is mainly because most games are themselves made for Windows operating systems, and with the advances in 2003 over 2000, the Windows server platform has become the most reliable and effective environment for our game servers.

Every miniscule detail is taken in to consideration when building our servers. The hard drives with game server files on them are kept separate from the hard drives containing the operating systems, providing a failsafe compartmentalization for purposes of data recovery, backup, and transfer. It also prevents the IDE/SCSI bus on which the game servers run from being slowed down by the operating system usage.

Awareness is everything.

“If you can’t see it, you can’t shoot it, so keep your eyes open.”

To host over 600 game servers is a very large task, and with this many game servers online, it is very difficult to monitor each one. Instead of attempting this, we began to monitor all of our physical servers. This is a much more scalable task, and one that is much more efficient. Through our network monitoring system we scan all of our physical servers (in real time) for the following:

· CPU usage

· Memory usage

· Hard drive seek time

· Current latency

· Packet loss

· Bandwidth usage

· System up time

· Hard drive(s) temperature

· Motherboard temperature

· CPU(s) temperature

Monitoring these items 24/7 gives us an accurate gauge of the load we are placing on each individual server. This also gives us an estimate of when we need to purchase new servers to prepare for more customers. This is where we developed our network baseline. This baseline was created to ensure our level of performance is kept at peek levels. If a system exceeds any part of our network baseline we have a clear signal that action is required to alleviate whatever part of the baseline was crossed. Our current baseline minimum specs for our systems are as follows:

Server Baseline

· Intel Xeon 2.4 GHZ dual processor

· 2 Gigs DDR PC-3200

· 80 Gig 7200 RPM hard drive

· Windows 2003 server

· 100/1000 Network Interface Card (NIC)

Network Baseline

· Cisco 2924 XL network switch

· Cat. 6 network cable

Performance Baseline

· Below 45% CPU usage

· 600 megabytes available system memory

· 6 gigabytes of available space on the operating system hard drive

· 40 gigabytes of space on the application (game server, voice server, file server) hard drive.

These are our minimum requirements for our servers and our network. As stated before, if any of these specifications are not met the equipment is either upgraded or liquidated and a replacement ordered.

With this working plan for our hardware, we had to come up with the same plan for bandwidth. For the most part (in the past and still today), dedicated server hosting companies bill by the gigabyte of data transferred each month. By that system, Art of War Central would have to pay for exactly how much data we transferred each month, which, as the demand for game servers grew, was quickly becoming a very large figure.

We were doubling our numbers each month. Although we were very excited to see our company doubling in size every month, the costs were increasing to match. Another bold maneuver was needed, and we came up with a plan to negotiate with the actual bandwidth providers, known as “backbone providers.” We would turn ourselves into a hosting company of a kind with Verio or Sprint. A bold move it was, too. We would have to create our own networks from scratch and start opening more locations in different areas of the country so our clients can get a good ping from any geographical location.

Over time we have used several different backbone providers. The problem was that many networks were not built to host games. Every hosting company was literally pushing us out of the door due to the extreme amount of load we were placing on their hardware. We kept searching, however, and eventually, we found our friend: Internap, with whom we are still allied to this day.

provide the quality of service we promise, we once again had to come up with something new.

Know your weaknesses and compensate.

“You are not perfect. But they don’t have to know that.”

Bandwidth is the majority of the cost. We have to ensure we have high quality bandwidth to provide top performance. If you have a team playing a scrimmage or in a league match you must be sure their game server is performing well. Using Internap as our bandwidth backbone was the way to go because of their unique and powerful routing technology.

Essentially, Internap bandwidth transit will always have the very best performing route possible to the end destination and is dynamically adjusted to route around any Internet backbone outages and problems due to the consistent regular performance probing, all resulting in the 100% uptime/no packet loss/sub 3ms latency SLA?. Value transit does not include the route probing and optimization on top of BGP?, and does not have an option of going over some of the largest tier 1 backbone transit links. Consequently, the right bandwidth carrier is the one most capable of handling mission-critical business and enterprise applications and implementations that require the lowest latency possible; this is why we have chosen to stick with Internap as our backbone provider.

Control the flow of battle.

“Dominate! This ain’t no spectator sport.”

Once we had the hardware costs down, the right network providers in many different locations, the demand for high performance game servers and the marketing in place to pull in over 150 game server orders a month, we created one of the most functional web pages on the internet to help us close the sales and to help the customers control their game servers. In the beginning no company provided a method to control their game server remotely. This was a problem for us too, for a while. Finally our programming staff sat down with the tech support, sales, and customers, to find out the needs and wants of all. We came up with a plan to provide an engine that will allow everyone to do what they need to do to make it faster, and easier.

Art of War Central announced SCMS (Server Control Management System). This system was requested by clients to offer:

· Start/Stop/Restart their game servers

· FTP straight from the interface

· File management section so they can easily edit configuration settings on the fly

Our tech support staff requested a long list which at the time was not all applied, yet in the next version our programming staff has assured us that all of this will work.

· Automated setup of game servers, so once a client orders a game server, the sales staff would just need to put in approval information and engine will auto install and setup the customer’s server.

· Improved support ticket system, the older version was a form customers would fill out tech support would have to find all the information first then, and didn’t keep records on hand so if customer was continuing to have same problem tech support didn’t know. The new system on the other hand, client puts in a support ticket, tech support gets it along with all customers information, and can read past tickets.

· Quick connect to the remote server. This would automatically connect to remote desktop to the client’s server, tech support time on a single ticket decreased average of 60 second on completing.

· The ability for customers to have and option of reopening ticket if they continue to have the same problem giving tech support a better understanding on what’s going on, even if it’s a new tech looking into the issue.

Sales had requested

· An online tracking system so they can see where each member was referred from. As the website logs do a good job to find out where the traffic is coming from, but even the biggest marketing deals, won’t actually provide customers. This has given our sales staff to target the areas better on where the actual sales come from. Our marketing staff has many advantages with this, as other companies are looking for the most traffic we are looking for more customers. Our sales team has lowered the cost on marketing expenses have also increased sales, and sales leads with this feature.

· Cancel request tracking system; During Art of War Centrals past we would keep an average customer 3-4 months. For many different reasons, customers have to leave at some point most with the same comment we will back, and they did 10% of the time. The average reasons to leave are money problems, team broke up, or trying another company. This information was hard to find out before the new system as our sales staff didn’t have this information. It was in a database though; sales staff demanded to know who is canceling why, and needed permission to contact the clients. Once the system was implemented sales staff will check cancellations daily, and have a chance to keep customers, and if not at least have the ability to show how much we care. We have increased our average customers staying to 5-6 months. Increased our recurring customers to 30%, most clients will find a new team, start there own, or move to a new game.

· The sales staff had also requested for there to be a chat system online so when our staff is not busy they can actually talk to visitors that are on the webpage. This feature is still beta and hard to do when you have so many people talking to you at once. Currently, its not used much, but the system works like a pop up ad, after the visitor clicks on at least 2-3 links on our site, this guy may be interested but might leave because the client didn’t find what it was looking for our, doesn’t know what he is looking for. The chat system will pop up the customer ask the client if he would like to talk to a sales rep, they say yes our staff is there at the other end to help the client, to find out what they might be looking for, and since this is beta still we don’t have hard facts, but each sales rep who has used it, has all agreed they helped close 2-3 sales on there shifts they used the chat system.

Many more features are in the works, to help staff, customers, and partners.

Learn as much from victory as defeat.

“A warrior is a student of anatomy. So shoot with your eyes open and listen for the loudest screams.”

Our customers pay for a game server that provides them a place to play 24 hours a day 7 days a week with a low ping and knowledgeable game administrator’s. They pay for the customer service they get by contracting for player-capacity.

Hardware and bandwidth usage is thus paid for by the client, month we use our profit to expand or upgrade hardware or to offer free services to the community, in order to improve and expand our public image & build the onling gaming community up and our position within the market.

Grow.

“He’s good. In fact, he’s the best. And yet…he could be better.”

We do grow. Not as quickly when Art of War Central was first formed, for at that time we were the only game server company up and running. Yet we still grow, and stand highest among the few companies that can support actual paid staff members, office, toll free numbers and many other differences about our company that put us on top. Many other providers are still trying to push the cost down for the gamer, and to do so they are pushing their staff numbers down, and pushing their performance down. Art of War Central is pushing performance, and customer experience, which is what the customers really want, up. The client will look for the lowest cost game server, but they tend to get tired of that after awhile, and we many of our customers from our cheaper competitors. Indeed, we find that those who leave us in favor of a cheaper solution come back within three months, their main reasons (by their testimony) being our superior customer support and performance.

With Art of War Central you are paying for a game server. Art of War Central worries about the hardware cost, and the cost of hardware investment is leveled over the entire customer base, preventing any one customer from getting…well…ripped off. With Art of War Central, we do that work for you and your clients. Art of War Central will make sure your servers are running at top notch performance and that you are not overloading any hardware for your game servers. With Art of War Central our staff is there 24/7 monitoring not only the hardware, but each game server on that hardware.

With Art of War Central there is no bandwidth cap. No limit

Art of War Central offers the better solution to your game server needs as a whole. Better by far. State-of-the-art, custom hardware, unlimited bandwidth, 24/7 customer support—these are just the beginning. Art of War Central’s game server hosting is second to none. Game Servers are our business. We have been hosting game servers since 2000, and around them we have built a business model that is flourishing. Our staff has the experience needed and no priority other than to provide beautifully performing game servers and the best support for those servers in existence.

High performance, low cost, professional support. The Art of your War, perfected.