Configuration
Sharding Configuration
Here's the list of things you can configure on the pod side:
role: the role of the current pod (defaults toRole.default)
Pod Roles
A Role is a tag (a simple wrapper around a String name) attached to each pod. Pods with the same role share a pool of shards, while pods with different roles are managed independently by the Shard Manager (each role has its own set of shards, its own rebalancing, and its own assignment state).
This is useful when you want to dedicate certain pods to certain entity types (e.g. a role for "game" pods and a role for "matchmaking" pods), or to scale parts of your system independently.
If you don't need this, just leave the default value: all pods will share the default role.
Note: a pod can only have one role. If a pod registers at an address already registered with another role, the Shard Manager considers the previous pod gone and unregisters it first.
numberOfShards: number of shards (for the pod's role)
How to choose the number of shards?
numberOfShards is used to calculate the shard ID from the entity ID. For that reason, it should be the same on all pods of a given role, and can not be changed while the app is running.
If this value is lower than the number of pods, some pods will have no shards (and host no entities), which is inefficient. If this value is too low, a difference of 1 shard between 2 pods will be quite significant and introduce some unbalance between the number of entities hosted on each pod. On the other hand, if there are too many shards, each pod will have a lot of shards, which will produce an unnecessary overhead.
A good rule of thumb is to set the number of shards to 10x the maximum number of pods that you expect to have.
selfHost: hostname or IP address of the current podshardingPort: port used for pods to communicate togethershardManagerHost: hostname or IP address of the Shard ManagershardManagerPort: port of the Shard Manager gRPC APIshardManagerTimeout: timeout for calls to the Shard ManagerserverVersion: application deployment version of the current pod (defaults to"1.0.0")
What's the version?
When performing a rolling update (upgrading all pods one by one without downtime), we want to avoid assigning shards to pods that will be stopped soon. Ideally, we want to move each shard only once during the whole process.
This serverVersion allows the Shard Manager to know which pods are old and which are new. It will then pick the new pods to assign shards.
Use one or more integers separated by dots, such as "123" or "1.2.3". All pods running the same release should use the same value, and newer releases should use higher values.
entityMaxIdleTime: time of inactivity (without processing any message) after which an entity will be stopped. It can be overridden for each entity type withEntityOptions.entityTerminationTimeout: time we give to an entity to finish processing its messages when it is stopped. After that, the messages still running are interrupted and their senders receiveEntityInterrupted. The whole stop, finalizers included, is bounded by twice this time. Rebalances wait for it. When a pod shuts down, its singletons then get this time to stop, after which they are left to finish in the background, so keep three times this value below your pods' termination grace period.sendTimeout: default timeout when sending a message, after which the sender receivesSendTimeout. It can be overridden for each messenger.refreshAssignmentsRetryInterval: retry interval in case of failure getting shard assignments from storage. Messages that can't reach their entity also make the pod read the assignments again, at most once per this interval, in case it missed a change.unhealthyPodReportInterval: interval to report unhealthy pods to the Shard Manager (this exists to prevent calling the Shard Manager for each failed message)simulateRemotePods: disable optimizations when sending a message to an entity hosted on the local shards (this will force serialization of all messages)unregisterRetrySchedule: retry schedule for unregistering the pod from the Shard Manager when it stopsentityInterceptor: anEntityInterceptor[AnyMessage]applied to the entities of every type, around the interceptor of each type (see Interceptors). It is a no-op by default.
Shard Manager Configuration
Here's the list of things you can configure on the Shard Manager side:
numberOfShards: a functionRole => Intreturning the number of shards for a given role (see above)
Why a function?
Because each role has its own pool of shards, you can pick a different number of shards for each role. If you only use the default role (or want the same number of shards everywhere), you can simply pass _ => 300 (or any constant).
apiPort: port to expose the gRPC APIrebalanceInterval: interval for regular rebalancing of shardsrebalanceRetryInterval: retry interval for rebalancing when some shards failed to be rebalancedpingTimeout: time to wait for a pod to respond to a ping requestpersistRetryInterval: retry interval for persistence of pods and shard assignmentspersistRetryCount: max retry count for persistence of pods and shard assignmentsrebalanceRate: max ratio of shards to rebalance in a single iterationpodHealthCheckInterval: interval for checking pod healthshutdownPersistTimeout: max time to wait for pods and shard assignments to be persisted on shutdown (10 seconds by default)
Rebalance Rate
The rebalance rate is there to prevent too many shards being assigned immediately to new pods. Instead of reaching a perfect spread right away, we can use several iterations to make sure new pods are able to handle the new shards. This is particularly useful if starting entities has a performance cost (e.g. loading state) and we don't want to start too many at once.
When a pod is leaving, its shards need to be immediately rebalanced so the ratio is not used in that case.
gRPC Configuration
When using shard-protocol-grpc, both pods and the Shard Manager require a GrpcConfig (GrpcConfig.default uses the defaults below):
maxInboundMessageSize: maximum size of a message received by the gRPC clients and servers (32 MB by default)executor: a custom executor given to grpc-java for its clients and servers (none by default)shutdownTimeout: time to wait for the gRPC server and channels to stop before forcing them (3 seconds by default)streamingPrefetch: number of messages fetched ahead of the consumer in streaming calls (16 by default)clientInterceptors: grpc-javaClientInterceptors applied to the calls made by pods and by the Shard Manager, e.g. to add headers for tracing.ShardingClientInterceptor.headersUpdaterbuilds one that updates the headers of each call. The last one runs first.serverInterceptors:ShardingServerInterceptors applied to the calls received by pods and by the Shard Manager, e.g. for logging or authentication.ShardingServerInterceptor.beforeEachbuilds one that runs an effect before each call. The first one runs first.