New to Searchlight? Please start here.

Installation Guide

Searchlight operator can be installed via a script or as a Helm chart. Install Searchlight

Using Script

To install Searchlight in your Kubernetes cluster, run the following command:

$ curl -fsSL https://raw.githubusercontent.com/appscode/searchlight/8.0.0/hack/deploy/searchlight.sh | bash

After successful installation, you should have a searchlight-operator-*** pod running in the kube-system namespace.

$ kubectl get pods -n kube-system | grep searchlight-operator
searchlight-operator-6945bcd777-4jdv7   3/3       Running   0          2m

Customizing Installer

The installer script and associated yaml files can be found in the /hack/deploy folder. You can see the full list of flags available to installer using -h flag.

$ curl -fsSL https://raw.githubusercontent.com/appscode/searchlight/8.0.0/hack/deploy/searchlight.sh | bash -s -- -h
searchlight.sh - install searchlight operator

searchlight.sh [options]

options:
-h, --help                         show brief help
-n, --namespace=NAMESPACE          specify namespace (default: kube-system)
    --rbac                         create RBAC roles and bindings (default: true)
    --docker-registry              docker registry used to pull searchlight images (default: appscode)
    --image-pull-secret            name of secret used to pull searchlight operator images
    --run-on-master                run searchlight operator on master
    --enable-validating-webhook    enable/disable validating webhooks for Searchlight crds
    --icinga-api-password          password used by icinga2 api (if unset, a random password will be generated and used)
    --enable-status-subresource    If enabled, uses status sub resource for Searchlight crds
    --enable-analytics             send usage events to Google Analytics (default: true)
    --uninstall                    uninstall searchlight
    --purge                        purges searchlight crd objects and crds

If you would like to run Searchlight operator pod in master instances, pass the --run-on-master flag:

$ curl -fsSL https://raw.githubusercontent.com/appscode/searchlight/8.0.0/hack/deploy/searchlight.sh \
    | bash -s -- --run-on-master [--rbac]

Searchlight operator will be installed in a kube-system namespace by default. If you would like to run Searchlight operator pod in searchlight namespace, pass the --namespace=searchlight flag:

$ kubectl create namespace searchlight
$ curl -fsSL https://raw.githubusercontent.com/appscode/searchlight/8.0.0/hack/deploy/searchlight.sh \
    | bash -s -- --namespace=searchlight [--run-on-master] [--rbac]

If you are using a private Docker registry, you need to pull the following 3 docker images:

To pass the address of your private registry and optionally a image pull secret use flags --docker-registry and --image-pull-secret respectively.

$ kubectl create namespace searchlight
$ curl -fsSL https://raw.githubusercontent.com/appscode/searchlight/8.0.0/hack/deploy/searchlight.sh \
    | bash -s -- --docker-registry=MY_REGISTRY [--image-pull-secret=SECRET_NAME] [--rbac]

Searchlight implements a validating admission webhook to validate Searchlight CRDs. This is enabled by default for Kubernetes 1.9.0 or later releases. To disable this feature, pass the --enable-validating-webhook=false flag.

$ curl -fsSL https://raw.githubusercontent.com/appscode/searchlight/8.0.0/hack/deploy/searchlight.sh \
    | bash -s -- --enable-admission-webhook [--rbac]

Searchlight 8.0.0 or later releases can use status sub resource for CustomResourceDefintions. This is enabled by default for Kubernetes 1.11.0 or later releases. To disable this feature, pass the --enable-status-subresource=false flag.

Using Helm

Searchlight can be installed via Helm using the chart from AppsCode Charts Repository. To install the chart with the release name my-release:

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update
$ helm search appscode/searchlight
NAME            CHART VERSION       APP VERSION DESCRIPTION
appscode/searchlight  8.0.0    8.0.0  Searchlight by AppsCode - Alerts for Kubernetes

$ helm install appscode/searchlight --name searchlight-operator --version 8.0.0 --namespace kube-system

To see the detailed configuration options, visit here.

Installing in GKE Cluster

If you are installing Searchlight on a GKE cluster, you will need cluster admin permissions to install Searchlight operator. Run the following command to grant admin permision to the cluster.

$ kubectl create clusterrolebinding "cluster-admin-$(whoami)" \
  --clusterrole=cluster-admin \
  --user="$(gcloud config get-value core/account)"

Verify installation

To check if Searchlight operator pods have started, run the following command:

$ kubectl get pods --all-namespaces -l app=searchlight --watch

Once the operator pods are running, you can cancel the above command by typing Ctrl+C.

Now, to confirm CRD groups have been registered by the operator, run the following command:

$ kubectl get crd -l app=searchlight

Accesing IcingaWeb2

Icinga comes with its own web dashboard called IcingaWeb. You can access IcingaWeb on your workstation by forwarding port 60006 of Searchlight operator pod.

$ kubectl get pods --all-namespaces -l app=searchlight
NAME                                    READY     STATUS    RESTARTS   AGE
searchlight-operator-1987091405-ghj5b   3/3       Running   0          1m

$ kubectl port-forward searchlight-operator-1987091405-ghj5b -n kube-system 60006
Forwarding from 127.0.0.1:60006 -> 60006
E0728 04:07:28.237822   10898 portforward.go:212] Unable to create listener: Error listen tcp6 [::1]:60006: bind: cannot assign requested address
Handling connection for 60006
Handling connection for 60006
^C⏎

Now, open URL http://127.0.0.1:60006 on your browser. To login, use username admin and password changeit. If you want to change the password, read the next section.

Configuring Icinga

Searchlight installation scripts above creates a Secret called searchlight-operator to store icinga configuration. This following keys are supported in this Secret.

Key Default Value Description
ICINGA_WEB_UI_PASSWORD changeit Password of admin user in IcingaWeb2
ICINGA_API_PASSWORD auto-generated Password of icinga api user icingaapi
ICINGA_CA_CERT auto-generated PEM encoded CA certificate used for icinga api endpoint
ICINGA_SERVER_CERT auto-generated PEM encoded certificate used for icinga api endpoint
ICINGA_SERVER_KEY auto-generated PEM encoded private key used for icinga api endpoint
ICINGA_IDO_PASSWORD auto-generated Password of postgres user icingaido
ICINGA_WEB_PASSWORD auto-generated Password of postgres user icingaweb

To change the admin user login password in IcingaWeb, change the value of ICINGA_WEB_UI_PASSWORD key in Secret searchlight-operator and restart Searchlight operator pod(s).

$ kubectl edit secret searchlight-operator -n kube-system
# Update the value of ICINGA_WEB_UI_PASSWORD key

$ kubectl get pods --all-namespaces -l app=searchlight
NAME                                    READY     STATUS    RESTARTS   AGE
searchlight-operator-1987091405-ghj5b   3/3       Running   0          1m

$ kubectl delete pods -n kube-system searchlight-operator-1987091405-ghj5b
pod "searchlight-operator-1987091405-ghj5b" deleted

Configuring RBAC

Searchlight introduces the following Kubernetes objects:

API Group Kinds
monitoring.appscode.com ClusterAlert
NodeAlert
PodAlert
Incident
incidents.monitoring.appscode.com Acknowledgement

Searchlight installer will create 3 user facing cluster roles:

ClusterRole Aggregates To Desription
appscode:searchlight:edit admin Allows admin access to Searchlight objects, intended to be granted within a namespace using a RoleBinding. This grants ability to create incidents manually.
appscode:searchlight:edit edit Allows edit access to Searchlight objects, intended to be granted within a namespace using a RoleBinding.
appscode:searchlight:view view Allows read-only access to Searchlight objects, intended to be granted within a namespace using a RoleBinding.

These user facing roles supports ClusterRole Aggregation feature in Kubernetes 1.9 or later clusters.

Using kubectl

# List all Searchlight objects
$ kubectl get clusteralerts,nodealerts,podalerts --all-namespaces
$ kubectl get ca,noa,poa --all-namespaces

# List Searchlight objects for a namespace
$ kubectl get clusteralerts,nodealerts,podalerts -n <namespace>
$ kubectl get ca,noa,poa -n <namespace>

# Get Searchlight object YAML
$ kubectl get podalert -n <namespace> <name> -o yaml
$ kubectl get poa -n <namespace> <name> -o yaml

# Describe Searchlight object. Very useful to debug problems.
$ kubectl describe podalert -n <namespace> <name>
$ kubectl describe poa -n <namespace> <name>

Detect Searchlight version

To detect Searchlight version, exec into the operator pod and run searchlight version command.

$ POD_NAMESPACE=kube-system
$ POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app=searchlight -o jsonpath={.items[0].metadata.name})
$ kubectl exec -it $POD_NAME -c operator -n $POD_NAMESPACE searchlight version

Version = 8.0.0
VersionStrategy = tag
Os = alpine
Arch = amd64
CommitHash = 9442863beb09a50a2c3818ab586fa5b1541fddf1
GitBranch = release-4.0
GitTag = 8.0.0
CommitTimestamp = 2017-09-26T03:00:58