Eskimo User and Administration Guide

Eskimo’s technical architecture can be illustrated as follows:

Three components are available in the storage layer:

The processing layer makes the following services available:

All Big Data / NoSQL middlewares as well as all User Interface (UI) Applications are operated by Kubernetes to achieve optimal cluster resources booking and negotiation.

The user layer is intended for data / result visualizations and platform administration with the following components:

Each and every software components is executed with Docker and packaged as a docker container. Runtime operation is ensured using Kubernetes for most services and some static services are handled with SystemD directly and defined as SystemD units.

A typical Eskimo application architecture can be illustrated as follows:

The above schema illustrates typical data flows within Eskimo

This is an example of a possible deployment of Eskimo on a 6 nodes cluster:

The Eskimo application itself can be deployed on any of the cluster nodes or on another, separated machine (as in the example above),

Requirements on machines to be used as Eskimo Cluster nodes are presented in the following sections:

Eskimo can also be used as a local Data Science laboratory, in which case the Eskimo backend is installed on the local user machine as well as all eskimo services. This is perfectly possible with Eskimo. In such case Eskimo wouldn’t be targeted towards large scale Big Data Analytics, but rather local Data Science experimentation or prototyping, make the user benefit from the Eskimo pre-packaged services (ElasticSearch, Spark, Flink, etc.)

Installing Eskimo on the local user machine is however tricky.
Eskimo does require indeed an IP address to identify the target node where eskimo services are to be installed.
A first idea one might have in this case is to use 127.0.0.1 (localhost) as single node target IP to proceed with the installation. Unfortunately, this doesn’t work as 127.0.0.1 resolves to different loopback interfaces in the various docker containers running eskimo services and as a consequence eskimo services are not able to reach each others when 127.0.0.1 is used as installation target.

So something else needs to be found as target IP address.

The best approach is to use the external interface IP address since in every possible configuration, this IP address will be made available from within Kubernetes PODs and native docker containers, just as services running on hosts themselves.

As stated in introduction, the eskimo backend can run on Microsoft Windows but in this case it’s only possible to download service container images from https://www.eskimo.sh. Building one’s own container images locally is not possible.

In addition, a property in the configuration file eskimo.properties needs to be adapted to a Windows environment, the properly that configures the path to the user definition file:

security.userJsonFile=/var/lib/eskimo/eskimo-users.json

needs to be changed to a folder existing on Windows and where the user running Eskimop has _write_access rights, such as e.g.

security.userJsonFile=c:/Windows/Temp/eskimo-users.json

Eskimo needs Java 11 or greater to run.

In addition, one needs to have either java in the path or the JAVA_HOME environment variable properly set in prior to starting eskimo with the provided startup scripts.

Use for instance the following commands on Linux:

(One might want to put the commands above in ones’s /etc/profile or /etc/bash.bashrc)

Use for instance the following commands on Windows:

(On Windows, one might want to define these as System Variables: Right-click on "My Computer", choose "Properties", then "Advanced System Settings", then "Environment Variables" and finally add or update the variables above as "System Variables")

In order to run eskimo, one needs to have

Eskimo is reached using a web browser (see startup logs). Supported web browsers are:

Note: there may be other browsers / versions supported (Safari, Opera, etc.) but they are not certified to work with Eskimo.

Network requirements with Eskimo are as follows:

In case of cluster deployment:

Linux’s distributions successfully tested for usage as Eskimo cluster nodes and officially supported are the following:

Other Debian-based or Red-Hat-based distributions could be supported as well but haven’t been tested so far and may require the administrator to adapt the setup scripts located in services_setup.

Eskimo performs some initial setup operations on every node of the cluster it needs to operate. Some of these operations would require Internet access to download dependencies (either RPM or DEB packages) if these are not properly installed in advance by administrators.

In case it is not possible to give access to internet to the nodes in the cluster one wants to operate using eskimo, one will find below the yum and apt commands used during nodes setup.
In case internet access from cluster node is not possible, one can reproduce these commands on one’s environment to find out about the packages that need to be installed in prior to have eskimo operating your cluster nodes:

Following commands are executed on a debian-based node:

Following commands are executed on a redhat-based node:

Following commands are executed on a SUSE node:

Again, if eskimo cluster nodes have no internet access in your setup, you need to install all the corresponding packages (those listed above and their transitive dependencies) before you can use these machines as eskimo cluster nodes.

After this initial setup is performed (in a process named Eskimo base installation and implemented by the script install-eskimo-base-system.sh), the eskimo installation is performed entirely without any need to access internet.

In case one wants to have Eskimo’s backend operated (automatically started, etc.) using SystemD, the script bin/utils/__install-eskimo-systemD-unit-file.sh can be used to perform all the required setup steps for a successful SystemD launch as well as installing the Eskimo SystemD unit configuration file.

Once extracted on the filesystem, the Eskimo folder contains the following elements:

Some command line utilities to ease eskimo’s administration are provided in bin/utils:

Building eskimo packages locally means building the services docker images on your local host machine running eskimo. This means that instead of downloading docker images from the eskimo repository, the user wants to build them on his own and only download the source package archives from their respective software editor web site (e.g. Apache, Elastic, etc.)

At any time after initial setup - and if and only if the chosen installation method is downloading packages, the user can apply setup again to check on the packages server (by default https://www.eskimo.sh) if updates are available for service docker images or kubernetes packages.

If you meet an error as the following one upon startup:

Eskimo uses a local file to define users and access credentials. Upon first startup, if that file doesn’t exist already, it is created by eskimo (with the default credentials above) at the path pointed to by the property security.userJsonFile in eskimo.properties.

If you experience the error above or something alike, change that property to point to a location where the first version of the file can successfully be created.

Public key authentication is a way of logging into an SSH/SFTP account using a cryptographic key rather than a password. This is a strong requirement in the current version of eskimo.

Keys come in pairs of a public key and a private key. Each key pair is unique, and the two keys work together.

These two keys have a very special and beautiful mathematical property: if you have the private key, you can prove your identify and authenticate without showing it, by using it to sign some information in a way that only your private key can do.

Public key authentication works like this:

You should generate your key pair on your laptop, not on your server. All Mac and Linux systems include a command called ssh-keygen that will generate a new key pair.

If you’re using Windows, you can generate the keys on your server. Just remember to copy your keys to your laptop and delete your private key from the server after you’ve generated it.

To generate an SSH key pair, run the command ssh-keygen.

You’ll be prompted to choose the location to store the keys. The default location is good unless you already have a key. Press Enter to choose the default location unless you already have a key pair there in which case you might want to take great care not to overwrite it.

Next, you’ll be asked to choose a password. Using a password means a password will be required to use the private key. Eskimo requires at all cost that you leave the password empty otherwise the key won’t be usable with eskimo - at least in this current version.
Press two times "Enter" there :

After that, your public and private keys will be generated. There will be two different files. The one named id_rsa is your private key. The one named id_rsa.pub is your public key.

You’ll also be shown a fingerprint and "visual fingerprint" of your key. You do not need to save these.

You can now SSH or SFTP into your server using your private key. From the command line, you can use:

If you didn’t create your key in the default location, you’ll need to specify the location:

If you’re using a Windows SSH client, such as PuTTy, look in the configuration settings to specify the path to your private key.

The ~/.ssh/authorized_keys file you created above uses a very simple format: it can contain many keys as long as you put one key on each line in the file.

If you have multiple keys (for example, one on each of your laptops) or multiple developers you need to grant access to, just follow the same instructions above using ssh-copy-id or manually editing the file to paste in additional keys, one on each line.

When you’re done, the .ssh/authorized_keys file will look something like this (don’t copy this, use your own public keys):

Once the above procedure properly followed and the public keys added to the authorized key for your the user to be used by eskimo, you can use the corresponding private key in the eskimo setup page to grand access to eskimo to the cluster nodes.

Whenever one wants to operate a cluster of a hundred of nodes with Eskimo, one doesn’t want to have to define the hundred nodes one after the other. Not to mention that wouldn’t make any sense since most nodes of that cluster would actually have the very same configuration (in terms of node native services topology).

This is the rationality behind the notion of "Range of nodes"- The idea here is to be able to add a single and unified configuration to all the nodes sharing the same configuration.

Single node configurations and range of nodes can be combined at will. Eskimo will however refuse to apply configuration if the resolution of the various ranges and single nodes leads to an IP address being defined several times.

Also, all nodes in a range are expected to be up and running and Eskimo will consider them so and report errors if one node in a range is not answering.
Should you have holes in your range of IP addresses, you are expected to define multiple ranges, getting rid of the holes in your range of IPs. This is fairly important if you want Eskimo to be able to manage your cluster without errors popping up frequently.

With all nodes from the cluster to be managed by eskimo properly identified (either as single node or as part of a range of nodes), services can be configured and deployed.

Some services are considered master services and are identified on the services selection window as unique services (understand services that can be deployed only once, e.g. Zookeeper, the Kube Master, etc.) and configured using a radio button

These "Master services" - considered unique - can only be configured in single node configuration and only once for the whole cluster:

Some other services are considered slave services and can be deployed at will, on one single or all nodes of the cluster (understand services that can be deployed multiple times, e.g. NTP, GlusterFS, Kube Slave, etc.) and configured using a checkbox on the services selection window.

These "Slave Services" - considered multiple - can be configured at will.

Once all nodes are properly configured with their desired set of services, clicking on "Apply Configuration" will initiate the Nodes Configuration process.

That setup process can be quite long on large clusters with plenty of nodes even though a lot of tasks are performed in parallel.

One should note that this configuration can be changed at will! Master services can be moved back and forth between nodes, slave services can be removed from nodes or added at will after the initial configuration has been applied, Eskimo takes care of everything !

As a sidenote, Eskimo Community Edition doesn’t support high availability for master services, one needs to acquire Eskimo Enterprise Edition for full high availability anf failover support.

Applying configuration is also useful when a service is reporting an error for instance such as needing a restart or being reported as vanished.
In such cases a first step to resolve the problem is getting to the "Configure Eskimo Nodes" screen and re-applying configuration.

Finally, whenever an installation or another operation fails, after fixing the problem (most of the time correcting the service installation scripts in the service installation framework), the installation or other operations can be recovered from where it failed by simply re-applying the configuration here.

Applying nodes configuration is re-entrant / idempotent.

The button "Force reinstall" enables the user to select services that will be reinstalled on every node from the latest service docker image available.
Dependent services will be properly restarted.

When mouse-over'ing a service on a node in the table view, the user has access to the service action menu which he can use to stop / start / restart a service or even force its full re-installation.

In addition to these default commands, Eskimo Services can provide additional custom commands made available to administrators and/or users in this action menu.

This is for instance the action menu when clicking on Zeppelin in the table view:

The menu "SSH Terminals" gives access to SSH terminals to each and every node configured in the eskimo cluster, just as a plain old SSH console, but from within your web browser.

As a design choice, the SSH Terminal doesn’t provide any toolbar but leverages on keyboard shortcuts to perform most useful actions.

SSH Terminal shortcuts:

Various notes related to Eskimo terminal console usage:

The Menu "SFTP File Manager" gives access to a web file manager which one can use to

Eskimo supports a Demo Mode which is in use for instance, on the DemoVM downloadable from the eskimo web site.
The purpose of the Demo Mode is to be able to showcase all possibilities of Eskimo - including administration features - while minimizing the runtime size and preventing users from breaking eskimo.

In Demo Mode, following actions are blocked:

Demo Mode is activated by changing the property eskimo.demoMode to true in the configuration file eskimo.properties:

The Eskimo DemoVM downloadable from the eskimo web site. It is intended as a demonstration of the features of the eskimo platform and enables users to test eskimo’s possibilities and feel it’s administrator and user experience.

The Eskimo DemoVM is provided with Demo Mode enabled by default, with the limits explained above (some actions are blocked).

In case a user wants to use the features that are disabled in Demo Mode, he needs to disable Demo Mode.

Note: the "host only" interface IP address of the DemoVM needs to be 192.168.56.41. Eskimo works by targetting cluster nodes using their IP addresses. In the case of the Demo VM, the targe cluster node is the very same VM itself. In this case, Eskimo is using the IP address of its own VM as installation target. So if the IP address of the DemoVM changes from 192.168.56.41 to anything else, Eskimo will interpret it as the target machine having disappeared.

In order to deactivate Demo Mode, change the property eskimo.demoMode back to to false in the confguration file eskimo.properties.

Unfortunately, this is not sufficient. The Eskimo DemoVM, for the sake of shortening it’s size, doesn’t package the Eskimo Service Package Images, it just packages placeholders instead.
So these placeholders need to be removed and the actual Eskimo Service Package Images need to be re-created or downloaded.

In order to do this, one should delete the content of the folder packages_distrib from the Eskimo installation folder:

When this is done the Eskimo Web UI will automatically bring the user back to the setup page and enable him to either build or download the Eskimo Service Package Images. Internet access from the VM is required.

Eskimo recommends to encrypt filesystem partitions use for data storage, either at hardware level if that is supported or at Operating System level.

Especially following folders or mount points have to be encrypted (depending on the services being installed)

It’s also possible within Eskimo to customize the ElasticSearch instances setup script to leverage on ElasticSearch’s native data at rest encryption abilities.

A note on user impersonation and user rights segregation: Eskimo Community Edition doesn’t support custom user rights segregation. All users within Eskimo Community Edition either have the user role - in which case they have access to business console and a few utilities - or the administrator role, who have full access to all Eskimo user and administration features.

If user rights segregation, authorizations enforcement and user impersonation are key concerns for one’s enterprise environment, one should consider upgrading to Eskimo Enterprise Edition which provides state of the art implementations of each and every Enterprise Grade requirement.

Some services leverage on SystemD to be managed and operated by Eskimo (node native services). Services themselves are implemented as docker containers.

This is how docker operations are mapped to systemctl commands :

Since every restart of a service creates actually a new docker container, containers are inherently not stateful and freshly restarted every time.
This is why the persistent data is stored under sub-folders of /var/lib which is mounted to the docker container.

But most services are Kubernetes services instead of node native (SystemD) services.

They can be managed using the Kubernetes Dashboard - which is prepackaged with Eskimo - or with the help of the kubectl command.

Eskimo provided services have kubernetes deployment descriptor generation scripts placed in /var/lib/eskimo/kube-services/ by eskimo.

Since software components and services within Eskimo are packaged as docker images, command line tools such as kafka’s create-producer.sh or spark’s spark-submit work only from within the respective kafka or spark executor docker container.

For this reason, eskimo provides for each of these command line tools a host-level wrapper in /usr/local/bin and /usr/local/sbin.
These wrappers take care of starting the required docker container and calling the corresponding command in it.

Even further, since most analytics services within Eskimo run on Kubernetes, these wrappers take care of tampering at startup with the container /etc/hosts file to dynamically resolve each and every service deployed on Kubernetes based on the Kube topology at command invocation time.

A specific generic shell wrapper is available in the form of the eskimo-kube-exec command located in /usr/local/bin.

This wrapper can be used to invoke a shell for instance with access to all Kubernetes services by their names with the following command call: eskimo-kube-exec bash.
This wrapper uses a generic container with a bunch of command line utilities available and is deployed with the kube-shell kubernetes service.

Master services that have a web console and other UI applications are wrapped and shown from within the Eskimo UI, in a consistent and coherent fashion, without the user needing to reach anything else that the Eskimo UI to access all services and features of an Eskimo cluster.

These wrapped UI applications are displayed as iframes in the Eskimo main UI window.

Whenever a service UI is being displayed by selecting the service from the menu, clicking the service menu entry a second time will force refresh the service iframe.

Now the remaining of this chapter presents each and every pre-packaged service:

The script zkCli.sh enabling an administrator to browse, query and manipulate zookeeper is available on the host running the zookeeper container as /usr/local/bin/zookeeperCli.sh

Eskimo approaches gluster shares management in a specific way.

First Gluster runs from within a docker container and is isolated from the host operating system.
Then Eskimo leverages on EGMI - Eskimo Gluster Management Interface - https://github.com/eskimo-sh/egmi - to manage and operate the cluster of gluster nodes.

The architecture can be depicted as follows:

EGMI is a daemon running on machines or containers alongside Gluster FS and taking care of managing gluster volumes and peers automatically (for most common operations).

The fundamental idea behind EGMI is that Big Data System administrators should not have to do so much manual operations to build and maintain a gluster cluster with its volumes and peers.

EGMI inspires from the way most widely used Big Data / NoSQL backends manage their nodes, shards and replicas transparently, balancing new replicas to new nodes automatically whenever a node goes down, etc. without an administrator needing to really worry about it.
EGMI aims eventually at bringing the same level of automation and reliability on top of Gluster FS and at simplifying most trivial aspects of gluster volumes management and repairing.

EGMI also includes a web interface for monitoring and to help administrators perform some simple manual operations and configuration.

Please refer to the EGMI page on github linked above for further informmation about EGMI.

Noteworthy details:

Gluster shares are mounted at runtime using standard mount command (fuse filesystem).

However eskimo provides Toolbox script that takes care of all the burden of mountint shared folders with gluster.

This Toolbox script is the available on cluster nodes at: /usr/local/sbin/gluster-mount.sh.
This script is called as follows:

where:

The beauty of this script is that it takes care of everything, from manipulating /etc/fstab to configuring SystemD automount properly, etc.

This script is related to the mount part (the client part) on hosts OSes running on the Eskimo cluster. A similar script is provided to run from within container to mount gluster shares from within containers (as required for instance for kubernetes operated services) : inContainerMountGluster.sh.
EGMI takes care of the GlusterFS backend management part.

Some notes regarding gluster usage within Eskimo:

The Kube Master takes care of deploying the CoreDNS POD and package.

The Kube Router is used for networking, firewalling and proxying on eskimo cluster nodes.

Both the Kube Master packages and the Kube Slave package takes care of mounting the gluster volume used to store the Kubernetes configuration to make it available to both master and slave processes.

All kubernetes system Docker images such as CoreDNS, Pause, etc. are packaged by Eskimo and deployed automatically.

Whenever logstash is distributed as a docker container, and yet to be used from other containers, such as Zeppelin, these containers can hardly (there are ways, but they are cumbersome) instantiate logstash processes. This is solved within Eskimo by leveraging on a command server and an always on container with the logstash software.

This command server is deployed as a Kubernetes StatefulSet in such a way that Kubernetes schedules this container on Eskimo cluster node.
The command server in these containers takes care of invoking logstash processes with the arguments passed to its API.

This works as follows:

logstash-cli supports all logstash arguments which are passed through to the invoked logstash instance within the logstash container.
In addition, it supports a non-standard argument that is specific to eskimo:

A specific package called logstash-cli packages the logstash-cli command presented above and makes it available on nodes where it is installed.

In addition to the command server / logstash-cli couple, a logstash command wrapper is provided that invokes logstash in an ad’hoc container created on the fly.

logstash-cli reaches the logstash instances by the kubernetes service name logstash.eskimo.svc.cluster.eskimo.

Nodes where logstash is installed automatically have the following gluster share created and mounted:

Eskimo is able to provision Kibana dashboards and referenced objects automatically at installation time.

In addition to the Kibana native samples distributed along Kibana, Eskimo provisions a sample Dashboard for Berka transactions used in Zeppelin sample notes.

A specific package called kafka-cli installs wrappers on the usual kafka command line programs usually bundled with kafka distributions. It is intended to be installed on nodes where operators, administrators or developers will interact with kafka.

A specific package called spark-cli installs wrappers on the usual spark command line programs usually bundled with Spark distributions. It is intended to be installed on nodes where operators, administrators or developers will interact with spark.

Nodes where spark is installed (either spark executor or spark history server or zeppelin) automatically have following gluster shares created and mounted:

The spark runtime is a registry only service. As long as no start job is running, there is no spark POD running in kubernetes. The spark driver takes care of instantiating spark executor as Kubernetes POD. The Spark driver itself can run within Kubernetes as a POD or outside Kubernetes as a standalone process.

The Spark History Server on the other hand, leveraging on the same container image as spark runtime PODs, is always up and running as a Kubernetes POD.

A specific package called flink-cli installs wrappers on the usual flink command line programs usually bundled with Flink distributions. It is intended to be installed on nodes where operators, administrators or developers will interact with flink.

The remote host to use in flink command line tools to reach flink deployed on Kubernetes with Eskimo is flink-runtime-rest.eskimo.svc.cluster.eskimo and the port is 8081.

Nodes where Flink is installed (either Flink App Master, Flink worker or Zeppelin) automatically have the following gluster shares created and mounted:

Within Eskimo, the pyflink python environment is available as a virtual environment packaged at the following location: /usr/local/lib/flink/opt/python/venv.zip.

In order for this virtual environment to be available to user submitted pyflink jobs, the following configurations must be declared:

or using Python DataStream API as following:

These configurations in the client job code are unfortunately required as of current version of eskimo.

A Flink POD is always running, it’s the Job Manager service which is constantly up and running and takes care of instantiating Task Manager PODs.

Within Eskimo, Zeppelin runs from within a docker container.
Command wrappers and custom command clients are available to enable it to use other services, running themselves as docker containers under eskimo.

In addition, zeppelin has access to shared folders used by the different services in order to be able to share data with them.
Following shares are mounted within the Zeppelin container:

These shared folders are automatically shared among the different nodes of the cluster using GlusterFS.

An additional share exist in order to be able to share data to the zeppelin docker container:

Zeppelin’s interpreters - such as the Spark interpreter wrapping the spark submit process or the ElasticSearch interpreter - can be instantiated globally for the whole zeppelin container of isolated per note.
Eskimo’s settings page enables an administrator to change this configuration globally for all zeppelin interpreters.

The default settings is shared which means that interpreters are shared by all notes within zeppelin.

Eskimo Enterprise Edition is required if one wishes to separate Zeppelin’s interpreters per user.

Upon Zeppelin installation, Eskimo sets up a set of Sample notes in Zeppelin to illustrate the behaviour of the Eskimo cluster using different frameworks and the different packaged technologies such as Flink, Spark, Logstash, etc.

These sample zeppelin notes are intended to demonstrate the possibilities with Eskimo and to show how Zeppelin can be used to program Spark batch jobs, Spark Streaming jobs, Flink jobs, etc.

The different sample note packages with Eskimo and available from within Zeppelin are described hereafter.

Within Eskimo, the packaging of prometheus is peculiar:

Eskimo CE provides two pre-packaged Grafana dashboards :

When a node running etcd is removed, either just before or right after removal, the node needs to be explictely removed from the etcd cluster.

Start by discovering the ID of the node by using etcdctl member list:

If, for instance, the node one wants to remove from the cluster is node4, then use its ID to remove the etcd node:

In the same way, if an etcd node is added to the cluster after the initial etcd setup, it needs unfortunately to be added explicitly to the etcd cluster before the eskimo cluster node running it is installed (or at least before the etcd service is installed on that node).

The command would be as follows:

When a node is reinstalled, the ID of the etcd service instance will be reinitialized. Even though both nodes names and peer URLs will be the same, as far as etcd is concerned, those would be two different etcd instances.

So one needs to first remove the previous instance and then add it back using both commands above.