amazon-web-services - Starting Mesos slave in Docker on Amazon Linux results in cgroup error - answerstu - answerstu.com answerstu

Starting Mesos slave in Docker on Amazon Linux results in cgroup error

I'm taking a docker-compose setup that's working on my Mac, and trying to get it running on a couple of AWS EC2 instances. The Mesos master Docker container came up fine, as did Zookeeper/Marathon, but the Mesos slave is having trouble:

$ sudo docker run --name mesos-slave1 -p 5051:5051 \
  -e "MESOS_LOG_DIR=/var/log" -e "MESOS_MASTER=zk://10.x.x.x:2181/mesos" \
  -e "MESOS_HOSTNAME=172.17.42.1" -e "MESOS_PORT:5051" \
  -e "MESOS_ISOLATOR=cgroups/cpu,cgroups/mem" -e "MESOS_CONTAINERIZERS=docker,mesos" \
  -e "MESOS_EXECUTOR_REGISTRATION_TIMEOUT:5mins" \
  redjack/mesos-slave:0.21.0
I0708 19:26:09.559125     1 logging.cpp:172] INFO level logging started!
I0708 19:26:09.569294     1 main.cpp:142] Build: 2014-11-22 05:29:57 by root
I0708 19:26:09.569327     1 main.cpp:144] Version: 0.21.0
I0708 19:26:09.569340     1 main.cpp:147] Git tag: 0.21.0
I0708 19:26:09.569350     1 main.cpp:151] Git SHA: ab8fa655d34e8e15a4290422df38a18db1c09b5b
Failed to create a containerizer: Could not create DockerContainerizer: Failed to find a mounted cgroups hierarchy for the 'cpu' subsystem; you probably need to mount cgroups manually!

I've done various searches, and have tried things like mounting /sys and similar approaches, with no luck. I've even tried running docker in privileged mode just as a sanity check. My docker-compose.yml specifies:

volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /usr/local/bin/docker:/usr/local/bin/docker
- /sys/fs/cgroup:/sys/fs/cgroup

And I started there, but mounting those on the EC2 instance doesn't work either. Since it works on my Mac, there's clearly some difference between OS X and Amazon Linux that's connected with the issue, but I haven't been able to determine a work-around. In case it's handy for OS identification purposes, on the EC2 instance it says:

$ uname -a
Linux ip-10-x-x-x 3.14.35-28.38.amzn1.x86_64 #1 SMP Wed Mar 11 22:50:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I may end up just installing Mesos directly on the EC2 instance, but it would be very convenient to avoid that by using the Docker container, of course.

If anyone's stumbled across this and found a solution, please share! I'll post back if I figure something out.

1 Answer

  1. Michael- Reply

    2019-11-13

    I was just about to post this question when I found one more thing to try, and it looks like it did it, so I figured I'd go ahead and post anyway after putting the work in, and hopefully the info may save someone else the time I spent trying to figure this out. Looks like the key is to mount /cgroup as a volume, which is presumably a difference between OS X and Amazon Linux (aka Centos variant). The final docker incantation that seems to be working:

    $ sudo docker run --privileged=true --name mesos-slave1 -p 5051:5051 \
      -e "MESOS_LOG_DIR=/var/log" -e "MESOS_MASTER=zk://10.x.x.x:2181/mesos" \
      -e "MESOS_HOSTNAME=172.17.42.1" -e "MESOS_PORT:5051" \
      -e "MESOS_ISOLATOR=cgroups/cpu,cgroups/mem" \
      -e "MESOS_CONTAINERIZERS=docker,mesos" \
      -e "MESOS_EXECUTOR_REGISTRATION_TIMEOUT:5mins" \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -v /usr/local/bin/docker:/usr/local/bin/docker -v /sys:/sys \
      -v /cgroup:/cgroup redjack/mesos-slave:0.21.0
    

    I'm still experimenting so I can't say for sure whether the privileged mode is required, and whether some of the other volumes are really needed (the docker stuff). But if this saves someone else some time, that's great.

Leave a Reply

Your email address will not be published. Required fields are marked *

You can use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>