Multiplayer

Czech technical University in Prague
Faculty of Information Technology
Department of Software Engineering
© Adam Vesecký, MI-APH, 2019

Literature

Networking architecture

Multiplayer

  • offers a capability for multiple players to act within the same game world at the same time
  • a very complex feature that has a significant impact on the game architecture, touching almost every fundamental structure

Multiplayer categories

  • single-screen multiplayer
  • split-screen multiplayer
  • networked multiplayer
  • MMOG

Multiplayer history

Local multiplayer games

  • Tennis for Two (1958)
  • Spacewar! (1962)

Networked multiplayer games

  • Empire (1973)
  • Maze War (1974)

Local area network games

  • Doom (1993)
  • Duke Nukem 3D (1996)

Online games

  • Quake (1996)
  • Unreal (1998)

MMO

  • Ultima Online (1997)
  • World of Warcraft (2004)

Issues

  • the main objective: how to synchronize several universes
  • all clients have to achieve a certain degree of synchrony
  • there is no (known) real-world picture for this type of problem
  • impact on the game design - game model, animation engine, sound engine, vehicle model, AI engine, physics,...
  • implement multiplayer features into a single-player game is a painful task
  • converting a multiplayer game into a single-player game is trivial
  • Naive approach: transfer a complete game state repeatedly to all clients
  • Most common approach: transfer a minimal subset of a game state that is required to reconstruct the complete information
  • Topologies: peer-to-peer, client-server

Peer-to-peer architecture

  • each device exchanges data with each other in a fully connected graph
  • used in Doom, early Command & Conquer, Age of Empires, Starcraft
  • given  peers, each peer must have  connections ->  in total
  • methods: single master, partial authority, full replication

Peer-to-peer architecture

Single master

  • one machine is chosen to be the game master
  • similar to client/server, but offers server migration upon disconnect

Partial authority

  • certain peers have authority over certain parts of the game
  • better bandwidth and processing power balance
  • difficult to implement and debug
  • each dynamic object belongs to exactly one machine
  • if one player drops out the game, all of the objects over which it had authority must be picked up by other machines
    • RSP (Referee Selection Problem)

Full replication

  • each peer executes the game logic and only events are exchanged
  • hard to synchronize to avoid alternate realities
  • difficult to get the same results (floating point calculations)
  • runs well for a few players over LAN but terribly over the internet

Peer-to-peer games

Doom (1993)

  • 14.4 kbps PPP or 28.8 kbps SLIP
  • each turn player inputs were exchanged with other peers
  • you had to wait for the inputs from the most lagged player
  • every 30ms, the input from each player is sampled and placed into a tic command (movement, strafing, turning, use, fire)
  • when the tic commands for all players have been received, the game advances its state

Age of Empires (1997)

  • for 8-player battle there could be up to 400 units
  • used Turn timer- queue for commands
  • AoE synchronizes the commands each player issued, rather than units
  • all commands during 200ms are saved into a buffer
  • when the 200ms are over, all the commands for that player's turn are transmitted over the network to all other players

Client-server architecture

  •  devices,  connections
  • server must handle  more messages per second
  • server quickly becomes the bottleneck (lack of power and bandwidth)
  • it is quite clear which code runs on a server machine
  • Dedicated server - only runs the game state and communicates
  • Listen server - server is an active participant in the game itself

Example: Quake

  • the first game that used partial reliability
  • each client acts as a dumb terminal
  • outputs are picked up by the prediction layer
  • the server runs at 20 FPS, while the client runs at 60 FPS
  • Command
    • the elementary unit of communication
    • used to update a position of the player, orientation, health,...
    • reliable commands (with impact on the game state) have to be confirmed

NetChannel header

Example: Starsiege tribes

  • sci-fi FPS, released in 1998
  • highly optimized networking model
  • supported up to 128 players over LAN or modem (56.6 kbps)
  • Data categories
    • non-guaranteed data: nonessential to the game
    • guaranteed data: guarantees both arrival and ordering (events)
    • guaranteed critical data: highest priority (player movement)
    • most recent state data: hit points

Example: Starsiege networking model

  • Game Simulation Layer
    • determines what particular client absolutely needs to know and what it ideally should know (need-to-know messages have higher priority)
  • Ghost Manager - replicates or  "ghosts" dynamic objects that are relevant
  • Move Manager - transmits movement data of the player as quickly as possible
  • Event Manager - maintains a queue of events generated by the simulation
  • Stream Manager - sends data to the connection manager
  • Connection Manager - manages a notification of delivery status
  • Platform Packet Module - socket API wrapper, implements reliability layer

Transport

Transport layer

  • TCP - Transmission Control Protocol
  • UDP - User Datagram Protocol
  • DCCP - Datagram Congestion Control Protocol
  • RUDP - Reliable User Datagram Protocol
  • SCTP - Stream Control Transmission Protocol
  • CUSP - Channel-based Unidirectional Stream Protocol (used in Irrlicht engine)
  • Well-known ports: 0-1023
    • 666: Doom
  • Registered ports: 1024-49151
    • 2302: Arma
    • 6112: Battle.net
    • 27015: Valve Servers
    • 27500: Quake
    • 28960: Call of Duty
  • Private ports: 49152-65535

Transport layer

  • TCP Header
  • UDP Header

TCP vs UDP

TCP

  • header size 20-40B
  • connection-based (requires 3 packets to set up a connection)
  • guaranteed reliability and proper ordering
  • may be prioritized over UDP packets
  • allocates a lot of resources to manage connections

UDP

  • header size 8B
  • lightweight and fast
  • no concept of connection
  • no guarantee of reliability or ordering
  • no flow control
  • may be dropped before TCP packets

TCP vs UDP

  • games have different requirements than applications
  • some data needs to be confirmed (actions, events), while other doesn't (updates)
  • some data has higher priority (player movement) than other (skin change)
  • UDP represents a basic streaming approach onto which developers can create any sort of a custom protocol that is most suitable for particular game
Using TCP is the worst possible mistake you can make when developing a multiplayer game.Glenn Fielder, 2008

Multiplayer engine architecture

  • the server is running the show and clients only process inputs and display results
  • Issues
    • each player has only partially consistent view of the game world
    • in some cases, the server may disagree with the client's state
    • there is a delay between taking an action an observing its result

Transfer control

Connection

  • when a client wants to join a game, it sends a hello packet to the server
  • once the server receives the hello packet, assigns a player ID and sends a welcome packet to the client, containing the assigned ID
  • when the client receives the welcome packet, saves its player ID and starts sending and receiving replication information to the server

Reliability

  • we have to identify packets by using sequence numbers and send them back to the server
  • example: ACK field in the message headers
  • issue: if the server sends 30 PPS and the client only sends 10 PPS, we need at least 3 ACKs included in each packet

Flow control

  • clients use basic congestion avoidance algorithms
  • servers can change streaming frequency based on the situation

Example: Message header

  • SEQ number - sequential number
  • ACK number - number of confirmed message
  • ACK bit array - confirmation flags for previous 32 messages
  • Type - type of the message (update, disconnect, command,...)
  • ActionID - id of particular action

Message types

Stream

  • doesn't need to be confirmed, contains a collection of continuous values
  • e.g. dynamic objects and their attributes (transformation)

Snapshot

  • complete information of the game state, sent either on demand or at given intervals

Command

  • messages that have an impact on the game state, have to be confirmed
  • example: UNIT_CREATED, UNIT_DESTROYED, BUILDING_COMPLETED

Action

  • high-priority messages (player's inputs, fire button,...)

Procedure Call

  • a generic message that allows to call any function (play sound, load asset, reset animation)

Connection messages

  • messages for handshake, ID assignment, disconnect etc.

Beacon

  • if the game lingers still (pause, no dynamic objects,...), the client should regularly send beacon messages, informing the server that the connection is still valid

Example: Goat attacks

Example: Goat attacks

Server-Client messages

  • GPCIdentifyPlayer
  • GPCReady
  • GPCAddPlayer
  • GPCRemovePlayer
  • GPCSpawnNPC
  • GPCRemoveNPC
  • GPCChatMessage

Client-Server messages

  • GPSRespawnRequest
  • GPSJoinRequest
  • GPSChatMessage
  • GPSBomb
  • GPSPlayerChanged

Serialization

  • a large game world can have hundreds of moving objects
  • we need to remove any information that doesn't need to be sent
  • messages should be as close in size to MTU (~1500B) as possible
  • average MMO packet size is only 30B
  • Naive approach
    • massive multiplayer battle
    • 5 players
    • 500 moving units
    • each unit has 20 attributes of 32bit size (transformation and state)
    • the server sends 30 messages per second
    • header size is 30B (240b)
    • required bandwidth:  Mbps

Serialization - binary footprint

struct Mage {

    int health;

    int mana;

}

 

void Serialize(const Mage* mage) {

    SendMessage(reinterpret_cast<const char*>(mage), sizeof(Mage));

}

 

void Deserialize(const Mage* output) {

    ReceiveMessage(reinterpret_cast<char*>(output), sizeof(Mage));

}

Health = 10, Mana = 14; Little Endian

Serialization - streams

  • streams allow us to customize what attributes to serialize and how

 struct Mage : public Streamable {

    vector<Item*> items;

    int health;

    int mana;  

  

    void SaveToStream(NetWriter* writer) {

        writer->WriteDWord(health);

        writer->WriteDWord(mana);

        writer->WriteDWord(items.size());

        for(auto item : items) item->SaveToStream(writer);

    }

  

    void LoadFromStream(NetReader* reader) {

        health = reader->ReadDWord();

        mana = reader->ReadDWord();

        int itemsCount = reader->ReadDWord();

        for(int i=0; i<itemsCount; i++) {

            items.push_back(new Item(reader));

        }

    }

}

Compression

Compression of bits

  • it is helpful to represent values with as few bits as possible
  • using floating point computations doesn't mean that we need all 32/64 bits
  • common technique is to examine the known range and precision

Entropy encoding

  • we compress data based on how unexpected it is
  • example: we can assume that the rotation is 99% of its time equal to zero, hence we can use only one bit to indicate this

Compression

Compression of attributes

  • we can serialize only attributes that vary
  • each object has a bit field that indicates which attributes follow in the stream

Delta messages

  • we store only differences between the current and the previous state
  • very efficient but sensitive to packet loss

Compression of the payload

  • Huffman encoding, run-length encoding, LZ4,...

Replication

  • the act of transmitting a state of an object from one device to another
  • each object must be uniquely identified (network ID)
  • the network message usually consists of a type of an object and all parameters required to construct this object on all machines

 switch(actionType) {

    case OBJECT_CREATED:

      int objectType = reader->ReadDWord();

      auto factory = creatorManager->FindObjectFactory(objectType);

      auto newInstance = factory->CreateInstance(reader); // parse parameters

      objects.push_back(newInstance);

    break;

    case OBJECT_DELETED:

      int objectId = reader->ReadDWord();

      sceneManager->removeObjectById(objectId);

    ...

  }

Latency

Latency

  • the amount of time between an observable cause and its observable effect
  • example: mouse click and a unit responding to its orders

Suitable latencies

  • FPS: 16-60ms
  • Virtual reality: < 20 ms
  • RTS: < 500ms

Non-network latency

  • input sampling latency (2ms)
  • rendering pipeline latency

Network latency

  • processing delay (encryption, routing)
  • transmission delay (time spent writing the bits to a medium)
  • queuing delay (router can only process a limited number of packets at a time)
  • propagation delay (information can't travel faster than the speed of light)

Example: Latency

  • both clients have the same latency

Example: Latency

  • client B has a higher latency

Synchronization

Interpolation

  • client can run at 60 FPS, the server usually sends state updates at 10-30 FPS
  • Interpolation: whenever the client receives a new state, it smoothly interpolates to that state
  • Problem: the object might change its interpolated value instantly (teleport)

Interpolation

Interpolation reserve

Extrapolation

Deterministic prediction

  • interpolation reserve is precise but delayed 1 cycle behind the server
  • precision of extrapolation depends on jittering of the extrapolated attribute
  • the client is ALWAYS at least 1/2 RTT behind the true state
  • some attributes can be handled on client side completely (camera pose in FPS and VR)
  • Prediction
    • the client predicts the future value by running the same simulation code
    • to perform extrapolation by 1/2 RTT, the client must approximate the RTT

Nondeterministic prediction

  • nondeterministic values are hard to predict (steering behaviors, other players' movement,...)
  • solution: make an educated guess and correct it when an update arrives
  • Dead reckoning - process of predicting a behavior of an entity based on the assumption that it will keep doing what it's currently doing

Server simulation              Client misprediction

  • Dealing with misprediction
    • Instant state update - update immediately when a correct value arrives
    • Interpolation - smoothly interpolate to the correct value
    • Second-order state adjustment - adjust acceleration instead of velocity

Server-side rewind

  • special case - long-range, instant-hit weapon
  • occurs due to the inaccuracies of dead reckoning and time dilation
  • server may change a state that has already been confirmed
  • Valve's Source Engine solution:
    • rewinds state on the server to exactly the state the player fired
    • server stores the poses of every relevant object for X last frames and looks up the two frames between which the client was interpolating

Wrong

Correct

Example: Server-side rewind

  • Client B has 3x higher latency than Client A
  • Client B performed a change before Client A did

Latency handling summary

Interpolation/extrapolation

  • smoothens out incoming values by interpolating to them

Deterministic prediction

  • runs simulated code, masks latency and keeps the client's state in sync

Dead reckoning

  • non-deterministic prediction
  • client uses the last known state of an object to extrapolate future state

Server-side rewind

  • the server buffers object positions for several frames to match the client's view when processing instant events

It is better to be wrong on time than right but late

Example: Source engine

  • LINK
  • server simulates the world at 30-60 FPS and sends 20 snapshots per second by default, using an interpolation reserve of 100ms
  • client samples inputs at 30-60 FPS and buffers snapshots for 100 ms
  • server uses time dilation and rewind

MMOG

  • Massively Multiplayer Online Game
  • Ultima Online, World of Warcraft, Guild Wars, Skyforge, Elder Scrolls Online
  • interaction is not really massive
    • players can only interact with a limited amount of players at the same time (~100)
    • when more than the expected number of players gather together, the game may crash

Swifty Invasion

Object Relevancy

  • an object is considered relevant for a client when it should be informed about its updates
  • in a MMOG, it may not be important to know about players several miles away
  • similar sort of problems as visibility culling (or spheres of influence)
  • Static zones
    • split the world into regions and assign a separate process/server to each region
    • only objects in the same zone as the player are relevant
  • Server partitioning
    • extension to the static zones, no direct interaction between players in two separate zones
    • WoW - each continent runs on a separate server process
  • Instancing
    • one shared game supports several separate instances at once
    • example: dungeons in WoW, Dota/LoL tournaments

Example: Ultima online

  • network is based on Pastry DHT (distributed hash tables)
  • world is split into regions of size based on the amount of players
  • each region is assigned an ID and mapped into the DHT
  • each node is responsible for exactly one region
  • 3 data types
    • static objects - immutable part of the world (terrain)
    • dynamic objects - interactive items and NPCs, deterministic
    • avatars - players, nondeterministic

Security

Classic threats

  • packet sniffing, phishing, man-in-the-middle
  • ghosting - scouting the players using multiple connections

Input validation

  • players can't perform an action that is invalid
  • only a client responsible for Player A can send an action that affects that player
  • client should validate the messages from server as well

Software cheat detection

  • actively monitors the integrity of the game
  • cheating SW can hook into the game, overwrite memory, modify files
  • map hacking - removing fog of war and revealing resources
  • bot cheat - bot that either plays the game or assists the player
    • e.g. WoW - dummy levelling up, aimbot in Fortnite,...
  • Valve Anti-Cheat - available for games that utilize Steamworks SDK
    • maintains a list of banned users, scans for known cheat programs
  • Warden - used by Blizzard

Libraries

Torque Network Library (opentnl)

ReplicaNet

  • powerful library for object sharing (sessions)
  • expensive

RakNet

NetStalker

Example: Unity multiplayer

  • uNet networking library, client-server topology
  • deprecated, new system is under development
  • very easy integration due to its component architecture
  • requesting values (unreliable and reliable), RPC, commands,...
  • NetworkTransform - synchronizes movement
  • NetworkAnimator - synchronizes animations
  • NetworkBehaviour - script for commands and RPC

Lecture 11 summary

  • Multiplayer: offers the capability for multiple players to interact within the same game world at the same time
  • Multiplayer categories: single-screen, split-screen, networked, MMOG
  • Topologies: peer-to-peer, client-server
  • Message types: stream, snapshot, command, action, procedure call, connection, beacon
  • Latency: amount of time between an observable cause and its observable effect
The client is ALWAYS at least 1/2 RTT behind the true state
  • Latency handling: extrapolation, prediction, server-side rewind
  • Security issues: scouting, map hacking, bot cheat

Goodbye quote

A famous explorer once said, that the extraordinary is in what we do, not who we are.Lara Croft, Tomb Raider 2013