Skip to content

Configuration ​

Sharding Configuration ​

Here's the list of things you can configure on the pod side:

  • role: the role of the current pod (defaults to Role.default)

Pod Roles

A Role is a tag (a simple wrapper around a String name) attached to each pod. Pods with the same role share a pool of shards, while pods with different roles are managed independently by the Shard Manager (each role has its own set of shards, its own rebalancing, and its own assignment state).

This is useful when you want to dedicate certain pods to certain entity types (e.g. a role for "game" pods and a role for "matchmaking" pods), or to scale parts of your system independently.

If you don't need this, just leave the default value: all pods will share the default role.

Note: a pod can only have one role. If a pod registers at an address already registered with another role, the Shard Manager considers the previous pod gone and unregisters it first.

  • numberOfShards: number of shards (for the pod's role)

How to choose the number of shards?

numberOfShards is used to calculate the shard ID from the entity ID. For that reason, it should be the same on all pods of a given role, and can not be changed while the app is running.

If this value is lower than the number of pods, some pods will have no shards (and host no entities), which is inefficient. If this value is too low, a difference of 1 shard between 2 pods will be quite significant and introduce some unbalance between the number of entities hosted on each pod. On the other hand, if there are too many shards, each pod will have a lot of shards, which will produce an unnecessary overhead.

A good rule of thumb is to set the number of shards to 10x the maximum number of pods that you expect to have.

  • selfHost: hostname or IP address of the current pod
  • shardingPort: port used for pods to communicate together
  • shardManagerHost: hostname or IP address of the Shard Manager
  • shardManagerPort: port of the Shard Manager gRPC API
  • shardManagerTimeout: timeout for calls to the Shard Manager
  • serverVersion: application deployment version of the current pod (defaults to "1.0.0")

What's the version?

When performing a rolling update (upgrading all pods one by one without downtime), we want to avoid assigning shards to pods that will be stopped soon. Ideally, we want to move each shard only once during the whole process.

This serverVersion allows the Shard Manager to know which pods are old and which are new. It will then pick the new pods to assign shards.

Use one or more integers separated by dots, such as "123" or "1.2.3". All pods running the same release should use the same value, and newer releases should use higher values.

  • entityMaxIdleTime: time of inactivity (without processing any message) after which an entity will be stopped. It can be overridden for each entity type with EntityOptions.
  • entityTerminationTimeout: time we give to an entity to finish processing its messages when it is stopped. After that, the messages still running are interrupted and their senders receive EntityInterrupted. The whole stop, finalizers included, is bounded by twice this time. Rebalances wait for it. When a pod shuts down, its singletons then get this time to stop, after which they are left to finish in the background, so keep three times this value below your pods' termination grace period.
  • sendTimeout: default timeout when sending a message, after which the sender receives SendTimeout. It can be overridden for each messenger.
  • refreshAssignmentsRetryInterval: retry interval in case of failure getting shard assignments from storage. Messages that can't reach their entity also make the pod read the assignments again, at most once per this interval, in case it missed a change.
  • unhealthyPodReportInterval: interval to report unhealthy pods to the Shard Manager (this exists to prevent calling the Shard Manager for each failed message)
  • simulateRemotePods: disable optimizations when sending a message to an entity hosted on the local shards (this will force serialization of all messages)
  • unregisterRetrySchedule: retry schedule for unregistering the pod from the Shard Manager when it stops
  • entityInterceptor: an EntityInterceptor[AnyMessage] applied to the entities of every type, around the interceptor of each type (see Interceptors). It is a no-op by default.

Shard Manager Configuration ​

Here's the list of things you can configure on the Shard Manager side:

  • numberOfShards: a function Role => Int returning the number of shards for a given role (see above)

Why a function?

Because each role has its own pool of shards, you can pick a different number of shards for each role. If you only use the default role (or want the same number of shards everywhere), you can simply pass _ => 300 (or any constant).

  • apiPort: port to expose the gRPC API
  • rebalanceInterval: interval for regular rebalancing of shards
  • rebalanceRetryInterval: retry interval for rebalancing when some shards failed to be rebalanced
  • pingTimeout: time to wait for a pod to respond to a ping request
  • persistRetryInterval: retry interval for persistence of pods and shard assignments
  • persistRetryCount: max retry count for persistence of pods and shard assignments
  • rebalanceRate: max ratio of shards to rebalance in a single iteration
  • podHealthCheckInterval: interval for checking pod health
  • shutdownPersistTimeout: max time to wait for pods and shard assignments to be persisted on shutdown (10 seconds by default)

Rebalance Rate

The rebalance rate is there to prevent too many shards being assigned immediately to new pods. Instead of reaching a perfect spread right away, we can use several iterations to make sure new pods are able to handle the new shards. This is particularly useful if starting entities has a performance cost (e.g. loading state) and we don't want to start too many at once.

When a pod is leaving, its shards need to be immediately rebalanced so the ratio is not used in that case.

gRPC Configuration ​

When using shard-protocol-grpc, both pods and the Shard Manager require a GrpcConfig (GrpcConfig.default uses the defaults below):

  • maxInboundMessageSize: maximum size of a message received by the gRPC clients and servers (32 MB by default)
  • executor: a custom executor given to grpc-java for its clients and servers (none by default)
  • shutdownTimeout: time to wait for the gRPC server and channels to stop before forcing them (3 seconds by default)
  • streamingPrefetch: number of messages fetched ahead of the consumer in streaming calls (16 by default)
  • clientInterceptors: grpc-java ClientInterceptors applied to the calls made by pods and by the Shard Manager, e.g. to add headers for tracing. ShardingClientInterceptor.headersUpdater builds one that updates the headers of each call. The last one runs first.
  • serverInterceptors: ShardingServerInterceptors applied to the calls received by pods and by the Shard Manager, e.g. for logging or authentication. ShardingServerInterceptor.beforeEach builds one that runs an effect before each call. The first one runs first.