BrandMeister Server supports several ways to make server clusters in a datacenter. In fact each cluster node is separate brandmeister process and can runs as a normal node but the main topic is how to reach zero-latency if we have more than one node close each to the next one.
We may have several reasons to think about clustering:
- Process of brandmeister reached performance of single CPU core. BrandMeister Server implements single-threaded event-based model ("Actor Model") to handle all actions. So the limit is a performance of a single CPU core.
- Security reasons. On some cases different masters managed by different administration teams can be hosted at the same DC as well as on the same hypervisor.
- Redundancy. No comments.
Running several processes on the same host
BrandMeister Server supports this option. This can solve a problem of reached performance of single CPU core. Each server process may utilize separate core. At the same time Tarantool and Mosquitto can be shared. You can balance load of different protocol stacks by this way, for example move "HyteraPTT RoIP" to separate process.
- Each node needs to have it's unique configuration file at /opt/BrandMeister/Instances with unique node ID (for example, network=25011 and network=250012)
- Each node have to be registered at systemd via setup.sh
- Each node have to be registered at global Registery Database
Reaching zero-latency
Regular transport for cummunications between nodes of BrandMeister Server is FastForward on top of UDP. This makes a traffic globally routed, but... it has no guaranies of delivery, packets can be reordered, we have to use jitter buffer.
How to have reliable delivery with zero latency if we can be sure of environment availability for this?
FastForward supports several options for that:
- Shared memory communications on the same host
- vSockets, when hosts are running at the same hypervisor (supported by KVM and vmWare)
- RDMA-based transport on top of InfiniBand, RoCEv1 or RoCEv2 when your hardware and DC has support of it
All these options can be used at the same time by defining records at the space OverriddenNodes at local Registry.
Ports
For all types of communications except shared memory FastForward uses unique numeric port number. It uses as:
- UDP and TCP port for IPv4 / IPv6 communications
- Datagram vSocket
- RDMA port number at TCP port space (reliable channel)
Configuration
Under Tarantool Registry:
- box.space.OverriddenNodes:replace({ <node id>, <key id>, <transport type>, <address>, <port> })
- Transport types:
- FORWARD_TRANSPORT_UDP
- FORWARD_TRANSPORT_RDMA
- FORWARD_TRANSPORT_VSOCKET
- Address:
- FORWARD_TRANSPORT_UDP and FORWARD_TRANSPORT_RDMA: host name, IPv4 address or IPv6 address
- FORWARD_TRANSPORT_RDMA: it could be InfiniBand GID in hexadecimal format with leading "0x", such as 0xfe8000000000000050800200008e4d39
- for FORWARD_TRANSPORT_VSOCKET: decimal CID number of host on hypervisor
BrandMeister