Myriad is a mesos framework designed for scaling YARN clusters on Mesos.
Myriad can expand or shrink one or more YARN clusters in response to events as per configured rules and policies.
The name Myriad means, countless or extremely great number. In context of the project, it allows one to expand overall resources managed by Mesos, even when the cluster under mesos management runs other cluster mangaers like YARN.
Myriad allows Mesos and YARN to co-exist and share resources with Mesos as the resource manager for the datacenter. Sharing resources between these two resource allocation systems improves overall cluster utilization and avoids statically partitioning resources amongst two separate clusters/resource managers.
Myriad is a work in progress.
- Support multiple clusters
- Custom Executor for managing NodeManager
- Support multi-tenancy for node-managers
- Support unique constraint to let only one node-manager run on a slave
- Configuration store for storing rules and policies for clusters managed by Myriad
- NodeManager Profiles for each cluster
- High Availability mode for framework
- Framework checkpointing
- Framework re-conciliation
The problem seems simple at first: How do clients determine the IP and port for a service that exist on multiple hosts?
When developing and running resource-efficient distributed systems like
Apache Mesos (a cluster manager) that simplifies the complexity of running applications on a shared pool of servers, this is a very important decision to make.
Jason Wilder has looked at a number of general purpose, strongly consistent registries (Zookeeper, Doozer, Etcd) as well as many custom built, eventually consistent ones (SmartStack, Eureka, NSQ, Serf, Spotify’s DNS, SkyDNS).
Many use embedded client libraries (Eureka, NSQ, etc..) and some use separate sidekick processes (SmartStack, Serf).
Interestingly, of the dedicated solutions, all of them have adopted a design that prefers availability over consistency.
Please read this really nice writeup by Jason Wilder to learn more.