Dev Story: Das Tal’s Backend & Exit Games’ Photon

This guest post was written by Sebastian Dorda, Technical Director and Developer for ‘Das Tal’ at indie studio Fairytale Distillery.

‘Das Tal’ is a fast-paced Sandbox MMORPG that focuses on player interaction. Full Loot and Open PvP create a game of ‘risk vs. reward’ where you weigh up your options before engaging others. PvP is never restricted, but cooperation is always rewarded. Each world is unique in its feature set and geography – and you pick those features! Time-boxed servers create winners and losers and allow you to regularly start anew. A class-less character system allows you to shape your character to fit your playstyle. You fight for resources to craft your equipment and construct your settlement. PvP combat is 100% skill-based, common and quick to be found. Both small-scale clashes and large-scale sieges are available for all players at any time.

Do not forget to sign-up as an alpha tester for Das Tal.

This time the topic is our backend. I will give an overview of our infrastructure and write in more detail about our game server which is based on Photon Server.

Be aware that this is just our way of doing it. If you have an opinion or comment about any of this I would be happy to hear from you.

A little background on the game design

Our game is designed so that we have lots of different small, isolated game worlds. We are talking about 1k – 2k registered players with 100-200 CCU (concurrently connected players) on one world. The reasons for the size are that we want to time-box worlds, have intimate player communities (within one world) and that we want to keep the technical complexity as small as possible.

The core gameplay is focused around a very fast and twitchy gameplay.
So we are somewhere in the middle between a bunch of Counterstrike servers and a “classical” WoW-like MMORPG structure with a huge world made of connected instances. Here is a gameplay example:

A birds-eye view on the infrastructure

This is an overview of the services that are somehow related to either the live operation of the game or its the development. Most of the services are already up and running although some only exist as dummies.

LIVE

• Account management (REST)
• Infos that survive a game world (names, achievements, …) (REST)
• Game world list (REST)
• Map file storage (HTTP, FTP)
• Patcher/client files (HTTP, FTP)
• Game servers (Photon Server)
• Client (Unity)
• Data collection (Graphite)
• Tracking (Honeytracks, Piwik, Google Analytics)
• Monitoring (icinga2)
• Website (Tumblr, web)
• Forum (web)
• Backup (Bacula)
• Newsletter (Mailchimp)

DEV

• Build/continuous integration/deploy server (Jenkins)
• Issue tracker (Hansoft)
• Version control (Git / Gitlab)
• Internal file sharing (Seafile)

A lot of these services are quite straightforward. Most MMO-like games have them – although probably in different tastes.

The game server

All these parts are important but the heart of all this is the game server. After evaluating different things we decided to use Photon Server. Mainly because we can easily share code between server and client (C#), because it is fast and well established. During pre-production I wrote a little bit about the game server middleware evaluation we did.

Overview

Our game server is a fully authorative server. The core game loop (simulation) runs at about 15 ticks/second. Because multi threading is incredibly hard we decided to limit the core game loop to just one thread. Designing and maintaining a single threaded game loop is much easier. There will, for example, be fewer strange timing bugs during high loads (like duplicated items).

The downside of this is that our core game loop can not scale horizontally and we will finally hit a point where we cannot scale-up one game world anymore. But our game worlds are small by design so that’s OK.

Other things like network IO, calls to external services, long-running calculations (pathfinding) or saving the world data run in different threads. Actually all these things run as a fiber using Photon’s internal fiber implementation.

We use a 2D grid index for client interest management. This is necessary to reduce traffic and CPU load to a manageable level.

Our gameplay is based on the entity system framework Artemis and it uses Farseer for collision and movement.

For small messages between client and server we only use Photon’s own serialization (key value list of basic data types). Bigger and more structured/complex data gets serialized by protobuf and encoded as a byte array in the Photon protocol.

The game state completely fits into memory. Therefore we just serialize it completly using protobuf and save it into a file. This is only feasible because each game world is small and limited in time and will not exponentially grow.

My tip: Next to the protobuf-encoded file we dump the complete game state as one big YAML file. This way we can easily inspect the game state with just a text editor. No extra tool needed.

All logging is done via log4net which is contained in the Photon Server. Windsor is our IoC container.

Our unit test setup uses the Visual Studio Unit Testing Framework and is based on the setup in the Photon MMO demo example.

Peer & service interaction

A peer represents a connected client and acts as a gateway to communicate with the player. Each peer has a fibre that processes the incoming and outgoing messages. Basically all peers exist in isolation. Communication between different peers is done using services. For example after finishing the authorization the peer registers at the chat service and can send and will receive chat messages in the future.

A peer has a protocol state object that represents its current state in the protocol. So if a client connects and joins the game it goes through these steps (from the view of the client):

• 1. Verify that the client version and the server version matches
• 2. Login and authorize the client (account service is involved)
• 3. Choose a character (account service is involved)
• 4. Get skill definition and other global balancing values from the server (balancing file, skill definition service is involved)
• 5. Get the map information necessary to download the static map data from an external server (download url, hash) (map storage service is involved)
• 6. Download the static map (if they are not already present on the client and the hash matches) (game server is not involved)
• 7. Instanciate the map (game server is not involved)
• 8. Enter the game(game service is involved)

Game service

The game service runs the fiber for the core game loop. It is the gateway between ingame (things running on the game fibre) and not ingame. So if a peer wants to enter the game it calls “gameService.Enter(peer)”. Character movement messages go through “gameService.OperationBodyMovement(peer, movement)”. All these commands get queued up in the game loop fiber and get executed between ticks (calculating one step of the game simulation).

Another job of the game service is to snapshot the current game state (synchronous) and hand it over to the map storage service which serializes and stores the state (asynchronous).

Core gameplay loop

This snipped shows how the tick method gets called within the game service.

public void Start()
{
    …
    // Calls tick each 10ms on the game fiber
    simulationTick = fiber.ScheduleOnInterval(Tick, 10, 10);
    …
}

private void Tick()
{
    // Tick gets called with a higher frequency than the 15 tick/s.
    // So call the realy simluation only if it is “behind”. Normally we run with 15 tick/s but
    // if there is server lag the simulation can catch up.
    if (isActive && tickTimer.IsProcessNecessary(world.Clock.Ticks, world.Clock.Delta))
    {
        …
        world.Clock.OneStep();
        world.TicksSinceLastRestart += 1;
        // artemis system update
        world.EntityWorld.Update();

        // fake lag to testing purpose
        if (serverConfig.GameFakeLagEachNTicks > 0)
        {
            if (world.Clock.Ticks % serverConfig.GameFakeLagEachNTicks == 0) System.Threading.Thread.Sleep(serverConfig.GameFakeLagTimeInMs);
        }
        …
    }
}

The heart of the game loop is the Artemis entity system. The entity system and all code that interacts with it is shared between client and server (the system that handles the game network synchronization is an exception). This is so the client can use the same data structures and methods if necessary. Currently the client only uses data structures and some check methods from the shared gameplay code. Artemis contains systems for the different gameplay parts (eg. ASCharacterRegs for health and energy regeneration, ASCharacterCast for casting skills).

A very important system is ASFarseer. It calculates collision and movement with the help of the Farseer library.

The ASSpatialIndex system contains different spatial indexes based on grids and is necessary for efficient client interest management. There are also special grid indexes to handle the footsteps (eg. protect regions against “footstep flooding”).

The ASNetwork (called “game network outgoing” in the diagram) handles all the game state synchronization with the clients.  This component has access to all the peer instances that are currently active within the game service.

The following steps show the interaction of these components on the basis of an example:

Example: A player moves her character and sees the result on the screen.

• 1. Client sends movement message
• 2. The peer receives the movement message
• 3. The peer’s ingame state processes the message and delivers it to the game service
• 4. After finishing the current tick the game service processes the movement command and plans the movement on the affected entity
• 5. During the next game tick the ASFarseer system simulates the movement of all entities
• 6. At the end of the tick the ASNetwork system sends back an ACK message for the processed input messages
• 7. Additionally the ASNetwork system sends relevant position updates to the active peers (using the grid)
• 8. The client adjusts the local predicted position to the received position

That is all for this week. If you have any questions about how we implemented things or if you have feedback then please get in touch with me at sebi@fairydist.com.