- Data Forwarder on an EKS Cluster—This method is recommended for ease of setup. Using an EKS cluster allows you to use EKS’s ability to associate an IAM role with a Kubernetes service account and configure your pods to then use that service account.
- Data Forwarder in any Other Kubernetes Cluster—This method aggregate data from non-EKS Kubernetes clusters including other cloud providers.
Deploying the Data Forwarder on an EKS Cluster
In this configuration, the Prometheus server collects cluster metrics from EKS nodes and the Prometheus Remote Write sends the collected metrics to an Amazon Managed Service for Prometheus (AMP) workspace. The data forwarder is deployed in an EKS cluster and connects to AMP to send container data to Densify. In this configuration, Prometheus collects metrics from the many clusters and the PromQL queries need labels to distinguish per-cluster data, so you must define each cluster name on the Prometheus server.Figure: Container Data Collection Using EKS-AMP
Figure: Container Data Collection Using EKS-AMP
Configuring the Data Forwarder on any Kubernetes Cluster
In this configuration, the Prometheus server collects cluster metrics from your container nodes and the Prometheus Remote Write Exporter sends the collected metrics to an Amazon Managed Service for Prometheus (AMP) instance. The data forwarder is deployed in a Kubernetes cluster (other than EKS) and connects to AMP to send container data to Densify. Prometheus collects metrics from the clusters your define and the PromQL queries need labels to distinguish per-cluster data, so you must define each cluster name on the Prometheus server. This is the same requirement for all mulit-cluster configurations.Figure: Container Data Collection Using K8S-AMP
Figure: Container Data Collection Using K8S-AMP
Configuring the Data Forwarder Using configmap.yaml
Use the configmap.yaml to provide connectivity details for Densify, Prometheus and identity details for all of the clusters to be analyzed. If necessary, refe to the table below, for details of the required settings.Table: Data Forwarder Settings in configmap.yaml
Table: Data Forwarder Settings in configmap.yaml
Term | Description | Value | ||
Forwarder Configuration | ||||
scheme | Specify the protocol to be used to connect to the Densify REST API interface. Select | https | ||
host | Specify your Densifyinstance details (e.g. myCompany.densify.com). You may need to specify Densify IP address. | <host> | ||
port | Specify the TCP/IP port used to connect to the Densify server. You should not need to change this port number. | 443 | ||
username<br/> | Specify the Densify user account that the Data Forwarder will use. This user must already exist in your Densify instance and have API access privileges. Contact [email protected] for the Densify user and epassword required to connect to your Densify instance. This user will be authenticated by the Densify server. | <user name> | ||
password | Specify the password associated with the user indicated above. Specify the password in plain text. | |||
encrypted_password | Specify encrypted password for the user indicated above. The password must be encryptedand supersedes any value that has been specified in the password field, above. Typically, [email protected] will provide a Densify username and corresponding encrypted password when they setup your Densify instance container data collection. Ensure that only one of the password options is enabled (i.e uncommented.) | <encrypted password> | ||
endpoint | This is the connection endpoint for the API. You can leave the default value. | /api/v2/ | ||
retry | The retry settings are optional and if not set specifically, then the default values are used. See Configuring Retry Connection Attempts for setting details. | |||
proxy | If you need to configure data collection through a proxy server, see Configuring a Proxy Host. | |||
Prometheus Configuration | ||||
scheme | This is set to https. Do not change the value or uncomment the setting. | https | ||
host | Specify the Prometheus hostname. In this configuration the data forwarder is not deployed in the same cluster as Prometheus, so you may need to specify a fully qualified domain name. i.e. https://<AWS region>.console.aws.amazon.com/prometheus/home?region=<AWS region>#/workspaces/workspace/<AMP workspace ID>
| <Prometheus host> | ||
port | Specify your Prometheus service connection port. The default port is 443. | 443 | ||
username | These parameters are not used for AMP configuration. You must use Sigv4, below. | <username> | ||
password | <password> | |||
bearer_token | <path> | |||
ca_cert | <path> | |||
§§TOOLTIPPRESERVEd379cb9991ae4a67812dc3cf3b914651§§, | This parameter is enabled. Do not change the value or uncomment the setting. See AMP-onboard-query-APIs for more details. If you are running on EKS using a service account with the appropriate IAM role, then you only need to specify the region. If you are not using this combination then you need to provide all of the following settings:
Optionally, if you are not running on EKS, you can create a secret and use it. In this case you only need to specify the region. See the readme on Densify’s github page for detailed instructions on configuring IAM roles for service accounts. | |||
retry | The retry settings are optional and if not set specifically, then the default values are used. See Configuring Retry Connection Attempts for setting details. |
| ||
Collection Settings | ||||
Indicate which entity types for which data should be collected. This section is optional and if not set or empty then all of the following entity types are included.
| true | |||
interval | Optional: Specify the interval at which to collect data. Select one of days, hours or minutes. If you make changes here these interval settings must correspond with cronjob settings. | minutes | ||
interval_size | Optional: Specify the interval at which data is to be collected. If the interval is set to hours and interval_size is set to 1, then data is collected every hour. These interval settings need to correspond with your cronjob settings. | 1 | ||
history | Optional: | 0 | ||
offset | Optional: Specify the number of days, hours or minutes (based on the interval value) to offset data collection, backwards in time. | 0 | ||
sample_rate | Optional: Specify the rate at which to collect samples within the specified interval. i.e. if the interval_size is set to 1 hour and sample rate is 5 seconds, then every hour 12 samples are collected. | 5 | ||
node_group_list | Optional: Specify a nodegroup label reference. By default, the “node_group_list” parameter is commented out and the data forwarder use the values, listed:
If you want to specify a node group label that is not included in the following list, uncomment this parameter and specify your node group label. See Configuring Node Group Collection for more details. |
| ||
name | In a multi-cluster configuration, each cluster must have a unique name as well as a set of unique identifiers for each cluster. | <name of first cluster> | ||
identifiers | Specify identifiers as a map of Prometheus labels (name and value) to uniquely identify the cluster. If you omit the identifiers, only one cluster can be present in this list. | <label name>: <label value> | ||
debug | Use this setting to turn debugging on/off. | <false> | ||
prefix | Specify a prefix for the compressed filename. | <zip file prefix> | ||