Spilt Milk on: Managing data effectively with SpatialOS in Lazarus

30 January 2017

lazarus - active missions

Spilt Milk Studios are currently working with Improbable’s SpatialOS to create Lazarus, a top-down multiplayer shooter set in a universe that can be developed, built up and then resets every week. Andrew Roper is their Technical Director.

All of Spilt Milk’s previous games were inspired by earlier media. Hard Lines was inspired by Snake, Tango Fiesta by ’80s action films. And Lazarus grew out of our love for Asteroids, one of the earliest arcade games.

Making Asteroids was not a hard problem. After all, Spacewar! – one of the earliest video games – was an Asteroids precursor, and was developed in six weeks from scratch on a PDP-1. Initially, the idea of developing an “Asteroids-like” was to get us used to SpatialOS and the workflow. We wanted something small in scope, as it was just three of us working on it for our week of platform ‘onboarding’ with Spatial.

It made sense from a lot of angles. Not only would most people understand its aim but it had simple systems that got more and more interesting the more we increased the scale and density of the simulation. It also meant that we didn’t have to concentrate on making brand new gameplay, but rather on the really juicy and brand new challenges and opportunities that SpatialOS offers.

lazarus map

Lost in Spatial

We approached the game by setting up a few rules for the gameworld and a few for the players. Then we started experimenting and feeling our way through additions, tweaks and changes based on a short-term plan and a long-term vision for the game. After doing this for a while, we stepped back and reassessed where we were heading.

We decided to take our learnings, and re-implement the core ‘fun’ of the game built around an ecosystem – something essentially, peculiar to SpatialOS. By having scale and persistence, we could build a really cool world with a closed loop of activity and its rules, agents and their goals, and then steer that towards something that would be fun to play. Data management proved to be one of the biggest challenges.

Considering our small team size, implementing the design without SpatialOS would have been a huge challenge to undertake.

Speed Data-ing

Moving towards that goal meant first dealing with all the data management issues an Asteroids game throws up. First, there were the asteroids themselves. In order to make sure they moved and flowed correctly we implemented client-side prediction and then took account of it in the FSim. We used a naval-style ‘dead reckoning’ system, so that once we know an asteroid’s heading and speed, we can predict where it will be at any given timestamp, and send server-side corrective messages periodically to correct it and and implement any changes.

Those ‘changes’ to an asteroid’s heading, of course include the effects of player action – primarily combat. Our first implementation of this had bullets originally acting as entities on the server. We soon realised that, considering the number of players that can be connected and the high rate of fire, this stacked up and became very expensive.

So we had to overhaul it. As Technical Director, my solution was to keep the bullets but remove the server-side entities, shifting them instead to local game objects. These exist separately on both the client and the FSim, to deal with damage/collision. This meant that now we send the controls of the player rather than the results, saving around 30 network messages/entity-handling events a second – a huge data-management change.

Finally, at the larger scale of our simulation, we needed to have dynamic territory control, which must remain manageable. We opted to have the equivalent of a server reset regularly in the game. This ‘Lazarus Wipe’ cleans up the territory efficiently and quickly, meaning there’s minimal downtime for players connected to the game during the event.

lazarus explosion

Example: Designing An Entity

For an example, here’s how I design an entity. Before anything gets started, I go through the following:

  • Where is the entity going to reside? Is it controlled by the FSim, the Player Client or by the GSim?
  • What is its functionality? Will it need to move around in the world?
  • What states does it need? Do these states already exist on another entity so I can re-use them, or do new states need to be created?
  • Which worker (GSim/FSim/client) should have authority over the properties of this entity? Do any behaviours need changing to move authority from one worker to another?

This procedure defines how I implement the blueprint of the nature, which then dictates how I set up the Unity Prefab. This is typically where more specialised code comes into play. There will be respective visualisers already in place for a lot of the states or behaviours (e.g. TakeDamage, Movement) which will get added on. Then the custom code for that entity is added, after which we iterate tests and refinement to make sure it’s functioning correctly. Thanks to how SpatialOS handles entity prefabs, this is a fairly quick iterative process.

Making sure that states were delegated to the correct owner, based on the entity, was an initial headache. The workaround for that was to write a piece of code which detected who the entity was owned by, and then to delegate states to the relevant authority. This was true in cases like taking damage environmentally, which is used by Asteroids, Players, Structures and so on.

I solved it with this piece of code:

def delegateDamage(): Unit = {
  val constraint = entity.watch[EntityOwner].ownerId.flatten.map(SpecificEngineConstraint)
  delegation = constraint match {
case None => entity.delegateState[EnvironmentDamageable](PhysicsEngineConstraint)
case Some(engine) => entity.delegateStateToOwner[EnvironmentDamageable]

} }

I don’t have to touch any networking code in order to make the world function; I’m purely updating state values and triggering an event on a state.

lazarus map v2

An Improbable Advantage

Considering our small team size, implementing this design without SpatialOS would have been a huge challenge to undertake. But with SpatialOS, we didn’t have to worry about the huge implementation of the backend and could just focus on the entities themselves.

SpatialOS’s Entity-Component-Worker (ECW) architecture helped too, as it’s so similar to Unity’s – it’s really just a case of picking from a set of states and behaviours for your natures, which in turn defines the blueprint for the entity that will exist on the server. It helped further my thinking of writing agnostic, decoupled code, and making sure that a state/behaviour doesn’t necessarily rely on another for functionality, but can be dropped in and out and still able to function without causing 20 bajillion errors.

Even now, I don’t have to touch any networking code in order to make the world function; I’m purely updating state values and triggering an event on a state. The number of enemies, asteroids, and entities in general that we can support right off the bat is crazy. Even in our first few days of using it, we were supporting thousands of entities with no sweat. Writing a system ourselves to do that? Well, we’d still be here writing it rather than writing this!