DataStore without losing progress: in-memory buffer, jitter and UpdateAsync

Almost every game that loses player progress loses it for the same reason: the code saves in the most direct way possible. SetAsync on every change, one store per system, a loop running every minute on every server at the same time. It works with ten players. With a thousand, it does not.

The official DataStore best practices page is long, but its core fits into four design decisions. This guide walks all four, with the reason for each — because a rule without its reason is the first thing abandoned under pressure.

The problem

The first mental adjustment is understanding what a DataStore is. The documentation offers the analogy: a DataStore behaves much like a database table. It is not a folder of files, and it is not a place where creating a new item is cheap.

From that comes the recommendation that contradicts the organising instinct: use a small and fixed set of stores, organising the records by key. The temptation to create one store per system — one for coins, one for inventory, one for achievements — produces a game that makes three requests where one would have done, and that cannot save the three together.

The same applies per player. Each player's persistent data should fit in one key, within the 4 MB per object limit. The documentation gives three reasons, and the third is the one that saves your night: that pattern reduces requests, allows related values to be updated atomically, and makes rollback easier. When a player's data is scattered, there is no "back to yesterday's state" — there is rolling back six pieces and hoping.

The rule that decides when to split is one sentence: whatever has to change together atomically stays in the same key. Currency and inventory change together in a purchase, so they live together. A telemetry statistic does not change together with anything, so it can move out.

Step by step

1. Name the key from something that does not change. The name should come from a stable identifier and a fixed pattern, like User_{UserId}. A display name is forbidden for the obvious reason — it changes, and the key is orphaned.

There are two less obvious gains. A fixed pattern makes the key predictable across servers and tools, which is what lets you write a support script that finds a player's data without scanning anything. And it also lets the automatic right to be forgotten processing identify that player's data — a legal obligation that either gets solved on its own, or does not.

To organise inside that, prefixes serve to group related keys. A game with several character profiles can use User_123456/Profile/Warrior and User_123456/Profile/Mage, and list them all by passing the common prefix. There is also the scope, which prefixes a string onto every key of that store instance — the default is global.

2. Decide about third-party libraries with your eyes open. The documentation does not forbid them: it says that in many cases one may be preferable to building from scratch. But it gives two criteria before adopting — review the owner, the maintenance and the features, and understand how to access and migrate the data without the library. That second one is what separates a dependency from a trap.

And it names one specifically: DataStore2 is classified as a legacy third-party library. For anyone already using it, there is an official tool in beta to migrate away from the one-store-per-player pattern it adopts.

3. Do not talk to the DataStore during the match. This is the decision that changes performance most. The recommendation is to keep a local copy on the server during play, and to update that copy instead of sending a request on every change.

Saving happens at four moments, and the official list is exactly that: periodically, when the player leaves, on server shutdown, and at critical points such as processing a purchase.

The periodic interval has two constraints. It has to fit within the request limits, and it has to be shorter than any session lock expiry — otherwise the lock drops with the player still playing. The official player data and purchase example uses 180 seconds.

4. Spread the requests out over time. A fixed-frequency loop starting together on every server produces a spike on every cycle. The recommendation has two parts: before starting the loop, each server or player gets a random initial offset; and in loops that do not require exact cadence, a bounded random jitter is added to each interval. Together they distribute the requests over time and reduce synchronised traffic spikes.

The artifact

The module below gathers the four decisions. One store, a key from a fixed pattern, a live profile in memory, saving with an initial offset and jitter, UpdateAsync with pcall and exponential backoff.

Two of its lines deserve attention. The first: when the read fails, the code does not create a default profile — it gives up on persisting for that session. Creating the default and saving later is how progress disappears, and the log stays clean. The second: the UpdateAsync callback returns nil when what is stored is newer than what we hold, which aborts the write instead of winning the race.

Common mistakes

The most expensive mistake is using SetAsync where UpdateAsync belonged, and it is expensive because it raises no error at all.

The official rule: prefer UpdateAsync() when the write depends on the current value or when several servers might write the same key. The reason is mechanical — UpdateAsync() reads the most recent value into your callback before writing, which reduces lost updates. SetAsync(), by contrast, overwrites the key without reading first, and can cause inconsistency if two servers write at the same time.

The scenario that produces is familiar to anyone who has seen it: the player is in two places — joined, left, joined again on a different server before the first one saved. The old server saves last, with stale data, and the last few minutes' purchase evaporates.

Notice that neither write failed. Both returned success, both appeared in the log, and the end result is the loss. It is the hardest bug format to believe when a player complains, because everything you have to look at says it worked.

The practical rule that comes out of that is short: SetAsync only when the write depends on nothing. Creating a new key, writing a value that entirely replaces the previous one without consulting it. Everything else — currency, inventory, progress, anything that adds or subtracts — is UpdateAsync. If you have to stop and think about which to use, the answer is already UpdateAsync.

The second mistake is the naive retry. The documentation asks for exponential backoff with jitter, a delay cap and an attempt cap — but it also asks for something almost nobody implements: process the retries in order for each key. An old request retrying after a newer one succeeded can overwrite the newer data. The retry becomes the bug.

And there is the case with no clean solution, only honest handling: a call that failed may have completed on the server anyway. "Failed" means the response did not arrive, not that the write did not happen. Code that assumes otherwise — and delivers the item again, or reverses the currency — creates the problem it thought it was avoiding.

Next step

Once the basics are standing, there is a case that demands measurement, not guesswork: the hot key. Each key has its own read and write limits, and a logical record that keeps hitting those limits after you have cut the unnecessary requests needs sharding.

The shard has to be deterministic — the documentation suggests something like User_{UserId}_Inventory_{ShardId}, derived from a stable identifier, so any server computes the same shard without coordinating with anyone.

The order matters and is easy to invert: sharding before reducing requests is optimising the waste. First the in-memory buffer, then the interval with jitter, and only then — if the limit keeps appearing — the sharding.

A five-minute exercise is also worth more than any refactor: list, on paper, every DataStore call your game makes and at what moment it happens. The list usually holds surprises — a GetAsync inside an interface function, a SetAsync fired by an event that happens several times a minute, a loop nobody remembers writing.

After that, the natural topic is errors and limits, which the documentation itself points to as the next reading. It tells you which failures are worth retrying and which are not — because insisting on an invalid-request error is spending quota to receive the same refusal, and that is how a game throttles itself with no real traffic at all.

-- ServerScriptService: the four official practices in a single file.
--
--   1. ONE store, key from a fixed pattern   -> PlayerData / User_{UserId}
--   2. in-memory copy, saved on an interval
--   3. random initial offset + jitter on the interval
--   4. UpdateAsync, with pcall and exponential backoff

local DataStoreService = game:GetService("DataStoreService")
local Players = game:GetService("Players")
local RunService = game:GetService("RunService")

local store = DataStoreService:GetDataStore("PlayerData")

local INTERVAL = 180 -- the one from the official example; below the session lock
local JITTER = 30 -- adds 0 to 30s so servers do not all fire together
local ATTEMPTS = 4

local DEFAULT = { coins = 0, level = 1, visits = 0 }

-- live profile: player -> in-memory table. Nothing reads the DataStore in the
-- middle of the match; everything reads and writes HERE.
local profiles = {}

local function keyFor(userId)
	return "User_" .. userId
end

local function copy(t)
	local new = {}
	for k, v in t do
		new[k] = v
	end
	return new
end

-- pcall + exponential backoff with jitter, capped attempts.
local function attempt(fn)
	local wait = 1
	for i = 1, ATTEMPTS do
		local ok, result = pcall(fn)
		if ok then
			return true, result
		end
		if i < ATTEMPTS then
			task.wait(wait + math.random() * wait)
			wait = math.min(wait * 2, 16)
		else
			return false, result
		end
	end
end

local function load(player)
	local ok, data = attempt(function()
		return store:GetAsync(keyFor(player.UserId))
	end)

	if not ok then
		-- do NOT create a default profile here: with no confirmed read, saving
		-- later would erase the real progress. Better to play without persisting.
		warn("read failed for " .. player.Name .. ": not saving this session")
		return nil
	end

	local profile = copy(DEFAULT)
	if type(data) == "table" then
		for k, v in data do
			profile[k] = v
		end
	end
	profile.visits = profile.visits + 1
	return profile
end

-- UpdateAsync, not SetAsync: the callback receives the most recent value, so
-- another server that wrote in the meantime is not erased unseen.
local function save(player)
	local profile = profiles[player]
	if not profile then
		return false
	end

	local ok = attempt(function()
		return store:UpdateAsync(keyFor(player.UserId), function(current)
			if type(current) == "table" and (current.visits or 0) > profile.visits then
				-- what is stored is NEWER than ours: do not overwrite
				return nil
			end
			return profile
		end)
	end)

	if not ok then
		warn("save failed for " .. player.Name)
	end
	return ok
end

Players.PlayerAdded:Connect(function(player)
	local profile = load(player)
	if not profile then
		return
	end
	profiles[player] = profile

	-- random initial offset: without it, every player who joined together
	-- saves together, forever.
	task.spawn(function()
		task.wait(math.random() * INTERVAL)
		while profiles[player] do
			save(player)
			task.wait(INTERVAL + math.random() * JITTER)
		end
	end)
end)

Players.PlayerRemoving:Connect(function(player)
	save(player)
	profiles[player] = nil
end)

-- server shutdown: the last save point on the official list
game:BindToClose(function()
	if RunService:IsStudio() then
		return
	end
	for player in profiles do
		task.spawn(save, player)
	end
	task.wait(3)
end)