Grafvision - Fotolia

Get started Bring yourself up to speed with our introductory content.

Build a successful Apache Mesos installation on Linux servers

As Apache Mesos gains traction for container orchestration, admins should try it out and evaluate its capabilities. Use this tutorial to set up the necessary software and avoid the pitfalls that plague the process.

Apache Mesos is an open source cluster management tool that abstracts and isolates resources within distributed IT environments. Enterprises use Mesos with, or as an alternative to, Kubernetes for container orchestration in large-scale deployments.

Mesos has a master-agent architecture, in which a master daemon manages agent daemons that run on a cluster node. In this tutorial for Apache Mesos installation, we set up a master on one server and an agent on a second -- but it is possible to run both on one machine. There is no difference in procedure; the only difference is where the master is located when you start the agent.

Readers should expect the build process -- compiling and linking the components of Apache Mesos -- to take about one hour on a two-core machine with 8 GB of memory.

First, avoid these pitfalls

Close any servers and end any running tasks on your machine before you begin compiling the Apache Mesos installation. This process can take 100% of the memory and prevent even SSH login attempts.

All commands must execute via the sudo command, which enables you to act as the administrative root user.

Test frameworks are not critical: It's a complicated process to write a Mesos test framework, and a regular user is unlikely to need one. Instead, IT admins are more likely to use a Mesos framework developed by an established vendor such as Hadoop, Spark or Cassandra.

Apache Mesos installation setup and prerequisites

Mesos runs on most Linux distributions, MacOS and Windows. This tutorial uses Ubuntu 16.04; installation instructions differ for MacOS and Windows.

Open firewall port 5050 on both the master and agent daemons. If you don't have the necessary permissions for that port, use a port that is normally open -- an online port scanner can check for those.

Run all the commands below. If you already have JDK 1.8, skip the step to install the Java development kit, in line two. Check for JDK 1.8 with the java -v command.

sudo apt-get install -y tar wget git

sudo apt-get install -y openjdk-8-jdk

 sudo apt-get install -y autoconf libtool

 sudo apt-get -y install build-essential python-dev python-six python-virtualenv libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev zlib1g-dev iputils-ping

Now download and unzip the Apache Mesos installation files. Store these files in the /usr/share folder, which is available to all users.

cd /usr/share/

sudo mkdir mesos

sudo wget http://www.apache.org/dist/mesos/1.7.2/mesos-1.7.2.tar.gz

sudo tar -zxf mesos-1.7.2.tar.gz

cd /usr/share/mesos/mesos-1.7.2

Build the software

You will need to build -- compile and link -- the software. Users can download a binary distribution, but at time of publication, the version number is several iterations behind what is available.

The simple build command, make, starts up the master and agent processes, and the following code should work: Installing the test frameworks that Mesos provides in Python and Java will probably fail.

sudo mkdir build

cd build

../configure

sudo make  

An option to speed up the Mesos installation is to use all of the available CPU on the machine and to turn off logging, by adding to the make command:

-j <number of cores> V=0

Install example frameworks

Apache Mesos does not run programs: It runs frameworks, which, in turn, run programs. Install the example frameworks. It will take about an hour to run.

sudo make check

Start Mesos

Complete the installation by starting up the cluster orchestrator. Use an IP address that is accessible from the internet to use a cloud provider and the browser interface: For example, Amazon Elastic Compute Cloud usually uses a 172.* address. Amazon networking uses network address translation to expose internal networks to the internet. Don't use the loopback interface 127.0.0.1, as that only works locally.

The next step in this Apache Mesos installation is to start the master daemon:

cd /usr/share/mesos/mesos-1.7.2/build/bin

sudo ./mesos-master.sh --ip=172.31.47.43 --work_dir=/var/lib/mesos

And then start the agent. This step can be done as many times as there are available agents. For a two-server setup, you could install the agent on both the operator server and the master to better emulate a cluster. Agent installation is exactly the same as that for the master, but starts a different program. To start the agent, assign to it the master's IP address and default port, 5050:

cd /usr/share/mesos/mesos-1.7.2/build/bin

sudo ./mesos-agent.sh --master=172.31.47.43:5050 --work_dir=/var/lib/mesos

Open the browser via the public IP address, for example: http://ec2-35-180-230-169.eu-west-3.compute.amazonaws.com:5050/#/

The browser dashboard displays all agents' statuses; here, one agent is running, and zero tasks are ongoing.

Apache Mesos browser dashboard
View status reports in the Mesos browser dashboard.

Test the framework

To test your Apache Mesos installation, configure it to run with a cluster-computing framework, such as Apache Spark. Or run any of the three commands listed below, which will configure the build as well, but they work only if the make check command returns a clean compile. The Python example requires Python version 2.7. Spark escapes that concern.

./src/test-framework --master=127.0.0.1:5050

 

 ./src/examples/java/test-framework 127.0.0.1:5050

 

python test_framework.py 172.31.47.43:5050

Establish high availability

High availability enables more than one copy of the master to run so that the system continues to work if a single master fails.

To set up Mesos in high availability mode, first run Apache ZooKeeper, a centralized service for distributed applications that helps maintain configuration information, synchronization and other operational aspects. Then start the masters and agents with the --zk argument: --zk=zk://host1:port1,host2:port2. This command alerts Mesos to the other master instances.

This was last published in June 2019

Dig Deeper on Application Maintenance on Production Systems

Join the conversation

1 comment

Send me notifications when other members comment.

Please create a username to comment.

How well did this installation tutorial work for you?
Cancel

-ADS BY GOOGLE

SearchDataCenter

SearchAWS

SearchServerVirtualization

SearchCloudApplications

TheServerSide.com

SearchCloudComputing

DevOpsAgenda

Close