SaltStack Config Enterprise Install, Multi-Master setup and Git repo configuration!

Featured

Hello Everyone! On today’s post, I’m going to do a step by step walkthrough of a SaltStack Config (from now on, SSC) enterprise install. The SSC enterprise install is meant to be used in production-grade environments, and can handle up to 5000 Salt Minions per master!

I will also cover how scale-out the deployment and add a new Salt Master (in a cluster configuration) to the deployment after it is finished.

In addition to this, I will cover how to configure a Git repo as a shared storage across masters.

Lastly, I will cover how to prepare a vRA Cloud Template to install the Salt Minion, configure it to use multiple masters, and run a Salt State on the deployment!

Let’s start! Buckle up!

Architecture

This deployment is going to have:

  • One VM for PostgreSQL and Redis – Required for persistent & in-memory database components. These two components could be also separated across two different VMs (and configured in HA, but only for manual failover).
  • One VM for RaaS (Responder as a Service) – This is going to be the GUI of SSC. The RaaS component can also be deployed in a cluster mode using an external load balancer, but I’m going to use a single one in this post.
  • Two VMs, one for each Salt Master, that will form a cluster. The secondary master and the cluster will be generated after the initial deployment.
  • Both Salt Masters will be configured to use a Github repository.

All VMs are running Centos 7 as the operating system.

Architecture

This is what my servers look like, from a vSphere point of view.

VMs

List of Steps

What do you need to do to carry out this deployment?

The first list of steps is based on this LINK, which is the ‘Installing and Configuring Saltstack Config’ Official Documentation from VMware.

  • Prepare the template for the VMs (you can prepare each VM separately but there are common items across all VMs)
  • Deploy the 4 VMs (take note of the IP addresses)
  • Prepare the VM that will become the Primary Salt Master
  • Prepare the VMs that will only be Salt Minions (the PostgreSQL / Redis & the RaaS)
  • Download & Copy the SSC installer to the Primary Master
  • Copy and edit the top state files
  • Edit the SSC settings Pillar file
  • Apply the highstates to the nodes
  • Install the License Key
  • Install and configure the Master Plugin on the Primary Salt Master
  • Log in for the first time and change the default credentials
  • Accept the Primary Salt Master key
  • Optional: Configure Backup for files (if not using a complete backup solution), Set up custom SSL certificates, SSO. (won’t be doing this as a part of this post) -> Link1, Link2, Link3

At this point, you’re going to have a functional SSC enterprise install, but you will only have a single master node. You still need to configure the Secondary Master, the cluster, and the repository!

The second list of steps is based on multiple sources and testing, since there isn’t a single source of information to configure all of this. Which is why I’m attempting to write this and condense it! I will be adding links throughout the steps.

  • Prepare the VM that will become the Secondary Salt Master
  • Copy the SSC installer to the Secondary Master
  • Prepare all the other minions to use Primary and Secondary Salt Masters
  • Copy Primary master key to the Secondary Master
  • Edit the RaaS configuration file (master plugin) on the primary master to add the Cluster ID
  • Install and configure the Master Plugin in the Secondary Master
  • Edit the RaaS configuration file (master plugin) on the Secondary Master to add the Cluster ID
  • Start Salt-Master on the Secondary Master and accept the Secondary Salt Master key
  • Install GitPython on both Salt Masters
  • Configure the GitFS filesystem and Github repository
  • Configure a Cloud Template on vRA to install the Salt Minion and configure the two masters
  • Create a Job using the Salt State hosted in Github
  • Run the Job!

Don’t be scared! I will be explaining every single step so you’re also able to have a successful deployment. Let’s start!

1: Prepare the template for VMs

You need to install OpenSSL, EPEL, and two libraries for Python (cryptography and OpenSSL) so this is what you need to run:

sudo yum install openssl -y
sudo yum install epel-release -y
sudo yum install python36-cryptography -y 
sudo yum install python36-pyOpenSSL -y 

You can also, disable firewalld (if you have any other firewalling solution) to avoid headaches with the inter-component communication.

sudo systemctl stop firewalld
sudo systemctl disable firewalld

After this is done, you can shutdown this template and use it to clone the 4 VMs needed in the deployment.

2: Deploy the 4 VMs

Self explanatory, deploy the VMs in your vSphere Environment.

3: Prepare the Primary Salt Master VM

You need to configure the Salt Repository, install the Salt Master and the Salt Minion.

sudo yum install https://repo.saltstack.com/py3/redhat/salt-py3-repo-latest.el7.noarch.rpm -y 

sudo yum clean expire-cache -y 

sudo yum install salt-master -y
sudo yum install salt-minion -y

Create a master.conf file in the /etc/salt/minion.d directory and add the following text to configure the minion to use itself as a Master.

master: localhost

Create a the minion_id file in the /etc/salt/ with a descriptive name for the minion using vi, in this scenario, for example, my Primary Master’s minion_id is ssc-gool-master1. This will be autogenerated based on hostname on the first run of the salt-minion service if its not set previously.

[centos@ssc-gool-master1 salt]$ cat minion_id
ssc-gool-master1

Enable and start the services

sudo systemctl start salt-master
sudo systemctl enable salt-minion
sudo systemctl start salt-minion

4: Prepare the PostgreSQL/Redis & RaaS VMs

You need to configure the Salt Repository, and Install the Salt Minion.

sudo yum install https://repo.saltstack.com/py3/redhat/salt-py3-repo-latest.el7.noarch.rpm -y 

sudo yum clean expire-cache

sudo yum install salt-minion -y 

Create a master.conf file in the /etc/salt/minion.d directory and add the following text to configure the minion to use the Primary Salt Master as its Master

master: IP_OF_MASTER

Set the minion_id (located in /etc/salt) to a descriptive name using vi, in this scenario, for example, my RaaS minion_id is ssc-gool-raas.

[centos@ssc-gool-raas salt]$ cat minion_id
ssc-gool-raas

My PostgreSQL + Redis minion_id is ssc-gool-psqlr

[centos@ssc-gool-psqlr ~]$ cat /etc/salt/minion_id
ssc-gool-psqlr

Enable and start the salt-minion service

sudo systemctl enable salt-minion
sudo systemctl start salt-minion

5: Download & Copy the SSC installer to the Primary Master

Download the SSC installer from Customer Connect (https://customerconnect.vmware.com/)

Copy the installer to a folder within the Primary Master (it can be the root directory as well), for example, /ssc-installer, and assuming our file is called ssc_installer.tar.gz.

scp ssc_installer.tar.gz USERNAME@IP_ADDRESS://ssc-installer/

Then extract the installer to a folder

tar -xzvf ssc_installer.tar.gz

6: Copy and edit the top state files

The top state files will be used by the orchestration to install the RaaS, Redis and PostgreSQL nodes.

At this point, you should take note of the Minion ID and the IP addresses of your three nodes, since you will be using them in the following steps. In my case, this is the information:

MINION ID: IP ADDRESS
ssc-gool-master1: 10.0.0.2
ssc-gool-psqlr: 10.0.0.3
ssc-gool-raas: 10.0.0.4

Now, you need to copy and edit the orchestration configuration files.

Important: The instructions below assume that this is a ‘greenfield’ Salt installation. If this is not the case, you might need to edit the following commands to work within your directory/folder structure.

Navigate into the sse-installer folder (this is the folder that was extracted from the tar.gz file) and run the following commands:

sudo mkdir /srv/salt
sudo cp -r salt/sse /srv/salt/
sudo mkdir /srv/pillar
sudo cp -r pillar/sse /srv/pillar/
sudo cp -r pillar/top.sls /srv/pillar/
sudo cp -r salt/top.sls /srv/salt/

In the /srv/pillar directory, you now have a file named top.sls. Edit this file to define the list of Minion IDs (not IP addresses) that you recorded previously. This is how it looks in my environment

{# Pillar Top File #}

{# Define SSE Servers #}
{% load_yaml as sse_servers %}
  - ssc-gool-master1
  - ssc-gool-psqlr
  - ssc-gool-raas
{% endload %}

Since as mentioned earlier, my 3 Minion IDs are ssc-gool-master1, ssc-gool-psqlr and ssc-gool-raas.

Also make sure that in the /srv/salt directory you also have a file named top.sls that looks like this:

base:

  {# Target SSE Servers, according to Pillar data #}
  # SSE PostgreSQL Server
  'I@sse_pg_server:{{ grains.id }}':
    - sse.eapi_database

  # SSE Redis Server
  'I@sse_redis_server:{{ grains.id }}':
    - sse.eapi_cache

  # SSE eAPI Servers
  'I@sse_eapi_servers:{{ grains.id }}':
    - sse.eapi_service

  # SSE Salt Masters
  'I@sse_salt_masters:{{ grains.id }}':
    - sse.eapi_plugin

7: Edit the SSC settings pillar file

You need to edit four different sections in the SSC settings pillar file to provide the values that are appropriate for the environment. These settings will be used by the configuration state files to deploy and manage your SSC deployment.

Navigate to the /srv/pillar/sse directory and edit the sse_settings.yaml file.

Section #1: Change the values of the four variables to match your Minion IDs. In my case, this looks like this:

# Section 1: Define servers in the SSE deployment by minion id
servers:

  # PostgreSQL Server (Single value)
  pg_server: ssc-gool-psqlr

  # Redis Server (Single value)
  redis_server: ssc-gool-psqlr

  # SaltStack Enterprise Servers (List one or more)
  eapi_servers:
    - ssc-gool-raas

  # Salt Masters (List one or more)
  salt_masters:
    - ssc-gool-master1

Section #2: Edit the following variables

  • pg_endpoint: use the IP address (or DNS name) of the PostgreSQL server. In my environment, this is 10.0.0.3.
  • pg_port: Port for PostgreSQL. In my environment, I left the default values
  • pg_username and pg_password: Credentials for the user that RaaS will use to authenticate to PostgreSQL

This section looks like this:

# Section 2: Define PostgreSQL settings
pg:

  # Set the PostgreSQL endpoint and port
  # (defines how SaltStack Enterprise services will connect to PostgreSQL)
  pg_endpoint: 10.0.0.3
  pg_port: 5432

  # Set the PostgreSQL Username and Password for SSE
  pg_username: salteapi
  pg_password: VMware1

Section #3: Repeat the previous steps but this time, to match your Redis parameters. Since we’re using the same server for both PostgreSQL and Redis, the IP will be the same.

# Section 3: Define Redis settings
redis:

  # Set the Redis endpoint and port
  # (defines how SaltStack Enterprise services will connect to Redis)
  redis_endpoint: 10.0.0.3
  redis_port: 6379

  # Set the Redis Username and Password for SSE
  redis_username: saltredis
  redis_password: VMware1

Section #4: Edit the variables that are related to the RaaS node

  • Since this is a fresh installation, do not change the eapi_username and eapi_password values. You will change the default password at a later step
  • eapi_endpoint: set it to match the IP address of your RaaS node. In my environment, this is 10.0.0.4
  • eapi_ssl_enabled: default is set to true. SSL validation is not required by the installer but it will be likely a security requirement in environments that use CA certificates.
  • eapi_ssl_validation: default is set to false. This means that the installer will not validate the SSL certificate.
  • eapi_standalone: default is set to false. This variable would be true in the case of the LCM install, in which all components are shared in a single node.
  • eapi_failover_master: default is set to false. This would be used if you were to configure a Multi Master configuration in failover mode (not active-active) and from within the installer. This will keep its default value since the scaling out will be done afterwards.
  • cluster_id: This variable defines the ID for a set of Salt masters when configured in a multi-master configuration. The default value should be left here, this will be edited at a later step, once the deployment is already running.

This is what my file looks like:

# Section 4: eAPI Server settings
eapi:

  # Set the credentials for the SaltStack Enterprise service
  # - The default for the username is "root"
  #   and the default for the password is "salt"
  # - You will want to change this after a successful deployment
  eapi_username: root
  eapi_password: salt

  # Set the endpoint for the SaltStack Enterprise service
  eapi_endpoint: 10.0.0.4

  # Set if SaltStack Enterprise will use SSL encrypted communicaiton (HTTPS)
  eapi_ssl_enabled: True

  # Set if SaltStack Enterprise will use SSL validation (verified certificate)
  eapi_ssl_validation: False

  # Set if SaltStack Enterprise (PostgreSQL, eAPI Servers, and Salt Masters)
  # will all be deployed on a single "standalone" host
  eapi_standalone: False

  # Set if SaltStack Enterprise will regard multiple masters as "active" or "failover"
  # - No impact to a single master configuration
  # - "active" (set below as False) means that all minions connect to each master (recommended)
  # - "failover" (set below as True) means that each minion connects to one master at a time
  eapi_failover_master: False

There is also a Section #5, but none of the values need to be edited at this step. These are the customer_id variable, which is a variable that uniquely identifies a SSC deployment, and the cluster_id variable, which will be edited once the deployment is already running and the scale-out is done.

8: Apply the highstates to the nodes

At this point, it would be wise to take snapshots of all your nodes, in case something goes wrong with applying the highstates, instead of having to troubleshoot a failed installation, it might be easier to rollback to the snapshot state and start over from this point.

Having said that, to apply the highstates, you need to do the following:

  • Accept the keys on your Primary Master, you can do that by running the command sudo salt-key -A which will accept all unaccepted keys (at this point, 3)
  • On your Salt Master, sync your grains to confirm that the Salt Master has the grain data needed for each minion. Since this is a fresh install, you can just run the command to target all the minions, which at this point are just 3.
sudo salt \* saltutil.refresh_grains
  • Then, run the following command to refresh the pillar data on all the minions
sudo salt \* saltutil.refresh_pillar
  • Lastly, run the following command to confirm the return data for your pillar is correct
sudo salt \* pillar.items

Confirm that the minions have received the pillar data that you edited on the sse_settings.yaml file, such as IP addresses, Minion IDs, etc

Now that you confirmed the data, it is time to apply the highstates to each node, by running the following command: sudo salt MINION_ID state.highstate – The PostgreSQL database should always be applied first

Which in my environment would look like:

sudo salt ssc-gool-psqlr state.highstate
sudo salt ssc-gool-raas state.highstate
sudo salt ssc-gool-master1 state.highstate

Confirm that the result of applying the highstates is succesful.

Note: you might get a ‘Authentication Error Occurred’ when applying the highstate to the Salt Master. This is expected, and it is deplayed because the Salt Master has not authenticated to the RaaS node yet. This will be solved at a later step.

If this has been successful, you now have a functioning install of SSC. But you still need steps to complete, let’s continue!

9: Install the License Key

To install the License key

  • Get your License Key from My VMware / Customer Connect (a vRA license is used)
  • Create a file with a filename ending in _license such as ssc_license for example
  • Edit the file and add your license key number.
  • Change ownership of the license file and copy the file to the /etc/raas directory
chown raas:raas ssc_license
mv raas.license /etc/raas
  • Restart the RaaS service: sudo systemctl restart raas

10: Install and configure the Salt Master Plugin

The Salt Master plugin allows the Salt Masters to communicate with SSC. The master plugin is installed on every Salt Master in your environment that communicates with SSC. At this step, you will only install it on the primary Salt Master

  • Log in to your Salt Master
  • The master plugin is located in the sse-installer/salt/sse/eapi_plugin/files directory. cd into that directory.
  • Install the Master Plugin by manually installing the Python wheel, using the following command, and replacing the exact name of the wheel file.
sudo pip3 install SSEAPE-file-name.whl --prefix /usr
  • Verify that the /etc/salt/master.d directory exists
  • Generate the master configuration settings
sudo sseapi-config --all > /etc/salt/master.d/raas.conf

Note: I had to do this step while being logged in as the root user since I was not able to generate the file even with a sudoer user. If this is your case, just switching to the root user and running the command will do the trick.

  • Edit the generated raas.conf file and update it to use your RaaS server
    • sseapi_server: Since you enabled SSL at a previous step, the URL should be https://IP_ADDRESS_OF_RAAS – in my environment, it would be 10.0.0.4
    • sseapi_ssl_validate_cert: However, since you’re not using a CA-signed cert, you should disable the validation to allow for the communication between the Master Plugin and RaaS

This file has more parameters that can be edited at this stage, for example, to set a custom certificate, or specific performance configurations. For more information, visit: LINK

  • Restart the Master Service: sudo systemctl restart salt-master

You can also check and edit the RaaS configuration file to edit RaaS related parameters. I won’t be covering them in this post and will be using the default values, but more information can be found at: LINK

11: Log in and change the default credentials

Log in to the SSC interface with the default credentials

Then go to Administration -> Authentication -> Local Users, and change the password for the root user

12: Accept the Salt Master Key

Go to Administration -> Master keys. You will see your Master node with a Key in the ‘pending’ state. Accept it.

At this point, you should see your minions pop up in the ‘Minions’ screen.

You can then run a simple command, such as test.ping to make sure that you can connect to your minions. For example:

Testing ping command from salt.master

You can also test this from the RaaS console, by selecting the minions and running the same job

Running test.ping job from the console

Congrats, if you made it here, you have a functioning distributed install of SSC! Now you will scale this out to allow for multiple masters and a shared repository!

A little break

At this point, we have configured our initial deployment. From now on, I will cover how to scale this out to add a secondary master, and configure a git repository!

14: Prepare the VM that will become the Secondary Salt Master

Follow Step #3 from the list, but this time using the Secondary Master VM!

15: Copy the SSC installer to the Secondary Master

Follow Step #5 from the list, but only copy the file, since you already downloaded it in that step.

16: Prepare all minions to use both masters

At this step, you need to edit the /etc/salt/minion.d in all the minions to use both masters. In the case of the nodes that are masters, you can just keep the localhost value and add the value of the secondary master to each node. In the case of the nodes that are minions (PostgreSQL/Redis, RaaS) you can append the IP of the secondary master to the file. Keep in mind that this will now become a list, and the syntax changes. The files should look like this (using the IPs from my environment)

  • For Primary Master:
[root@ssc-gool-master1 centos]# cat /etc/salt/minion.d/master.conf
master:
  - localhost
  - 10.0.0.5
  • For Secondary Master:
[centos@ssc-gool-master2 pki]$ cat /etc/salt/minion.d/master.conf
master:
  - localhost
  - 10.0.0.2
  • For the PostgreSQL/Redis & RaaS nodes:
[centos@ssc-gool-raas /]$ cat /etc/salt/minion.d/master.conf
master:
  - 10.0.0.2
  - 10.0.0.5

Restart the Salt Minion service on each node after editing the files: sudo systemctl restart salt-minion

17: Copy the Primary Master Key to the secondary master node

This is a requirement to be able to use a redundant master (regardless of it being configured in an active-active configuration or active-passive). The masters need to share the private and public key.

You should log in to the Primary Master and run the following commands to copy the files. This can be done with the root user if you run into any issue with accessing the folder. Overwrite the existing files if prompted.

cd /etc/salt/pki/master
scp master.pem USERNAME@IP_OF_SECONDARY_MASTER:/etc/salt/pki/master/
scp master.pub USERNAME@IP_OF_SECONDARY_MASTER:/etc/salt/pki/master/

Then, log in to the Secondary Master and restart the Master Service: sudo systemctl restart salt-master

18: Edit the RaaS configuration file (Master Plugin) on the primary master to add the Cluster ID

Since we’re going to place both masters in the same cluster, we need to make RaaS aware of this. This configuration is handled in the Master Plugin configuration.

To do this change, open the /etc/salt/master.d/raas.conf file on the Primary Master and edit the value of the sseapi_cluster_id variable. In my environment, this looks like this:

sseapi_cluster_id: goolcluster                                         # SSE cluster ID for this master (optional)

As we saw on the Architecture image, my cluster will be called goolcluster

Then, restart the Salt Master service on the Primary Master: sudo systemctl restart salt-master

19: Install and configure the Master Plugin in the Secondary Master

Follow Step #10 using the Secondary Master VM.

20: Edit the RaaS configuration file (Master Plugin) on the secondary master to add the Cluster ID

Follow Step #18 using the Secondary Master. Make sure to use the same Cluster ID, in this case, goolcluster.

21: Start Salt-Master on the secondary master and accept the second Salt Master Key

Log in to the Secondary Master, then run the following command to start the Salt Master service: sudo systemctl start salt-master

Then, log in to SSC using a browser, and accept the Master Key for the secondary master. Once it is accepted, it should look like this:

Both master keys accepted

At this point, since all minions are configured to use both masters, you will get pending minion keys in the SSC console. You can accept them there.

Note: Since both masters are part of the same cluster, SSC will recognize them as the same node. They will show up in the ‘Pending’ view of keys, but they will show up as already accepted. You need to accept them again (since this will be accepting the key in the secondary master). In the future, new minions will only need to be accepted once, since accepting on SSC will run the job to accept keys on both Salt Masters

The next steps will be about configuring the GitFS filesystem, which is one of the ways you can use a shared filesystem for both masters. It is a requirement to have a shared filesystem across masters, otherwise you could have inconsistent information depending on what master is being used for any given Job instance.

22: Install GitPython on both Salt Masters

Configuring a GitFS filesystem in a Salt Master can be accomplished through two methods:

  • Via GitPython
  • Via pygit2

I was having trouble with pygit2 in CentOS and getting Salt to recognize the version (this has been reported in multiple Git and Stackoverflow posts), so I ended up using GitPython instead, and this is what I will be describing.

To install GitPython, and its dependencies (such as the git cli), log in to both Salt Masters and run the following command: sudo pip3 install GitPython

After installing it, make sure that it shows up as being used by Salt. Salt uses its own Python version so some packages don’t always get recognized.

Run the following command: salt -V

[centos@ssc-gool-master2 ~]$ salt -V
Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.9.1
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: 4.0.9
     gitpython: 3.1.20
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: 3.14.1
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: 5.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

Salt Extensions:
        SSEAPE: 8.6.2.11

You can see that GitPython shows up as installed and with version 3.1.20, while pygit2 is not installed.

23: Configure the GitFS filesystem and Github Repository

The URL for the repository that I’m using now is public, and it is https://github.com/luchodelorenzi/saltstack -> you can use it as well for doing the same tests I will be doing on this deployment.

All the steps of this configuration need to be done on both Salt Masters, since they’re now running in a cluster

  • Edit the fileserver_backend parameter and add the ‘gitfs’ filesystem to the /etc/salt/master.d/raas.conf file.
# Enable SSE fileserver backend
fileserver_backend:
  - sseapi
  - roots
  - gitfs
  • Create a new file in the same directory, called gitfs.conf and add the following parameters
gitfs_provider: gitpython

gitfs_update_interval: 60

gitfs_base: main

gitfs_remotes:
 - https://github.com/luchodelorenzi/saltstack.git

Note: the same raas.conf file could have been used to append the gitfs parameters. However, Salt will look for all *.conf files in the master.d directory, so separating this in a different file could make it easier to maintain / check.

What do each of the parameters mean?

  • gitfs_provider: The provider that will be used to leverage GitFS. In this case, GitPython
  • gitfs_update_interval: Update interval for gitfs remotes.
  • gitfs_base: Defines what branch or tag is used as the base environment. The main branch on my repository is main, but it can change depending on yours
  • gitfs_remotes: List of the repositories. Only adding one in this deployment. You can have multiple deployments, and some parameters are overridable per remote.

There are multiple other parameters for GitFS. For more information please follow this LINK on the GitFS section. You can also follow this other LINK for a GitFS walkthrough.

After configuring this, restart the Salt Master on both nodes, by running the following command: sudo systemctl restart salt-master

Now, you need to check if our files are being read from Github! Since the mapping was done to the base environment, running the following command will show every state file in that environment. For example:

[centos@ssc-gool-master2 master.d]$ sudo salt-run fileserver.file_list saltenv=base
- _beacons/status.py
- apachenaming/init.sls
- presence/init.sls

The apachenaming state resides on Github, as you can see here:

Github Repository

So hooray, we can now use this Salt state from Github, and this is shared across both masters, has version control, you can use multiple repos, multiple branches. Pretty cool, isn’t it?

24: Configure a Cloud Template to Install the Salt Minion and Configure Two Masters

Since the vRA integration with SSC does not allow to integrate with Multiple Masters, I will use CloudConfig to perform the initial installation and configuration. As a prerequisite for this, your template should be prepared to use CloudConfig

This is the CloudConfig Code in the Template I’m using:

cloudConfig: |
        #cloud-config
        hostname: ${self.resourceName}
        runcmd:
          - curl -L https://bootstrap.saltstack.com -o install_salt.sh
          - sudo sh install_salt.sh -A 10.0.0.2
          - sudo chown ubuntu /etc/salt/minion.d
          - sudo rm /etc/salt/minion.d/99-master-address.conf
          - sudo echo -en 'master:\n  - 10.0.0.2\n  - 10.0.0.5' > /etc/salt/minion.d/master.conf
          - sudo systemctl restart salt-minion

What am I doing here?

  • Download the install_salt.sh bootstrap script
  • Run it to install the Salt Minion and point it to one of the masters
  • Change ownership of the minion.d directory to the ubuntu user which is the user that’s being used by CloudConfig in this template
  • Remove the 99-master-address.conf that was generated during the Salt Minion install
  • Create a new file with the addresses of the two Salt Masters (and using the correct list syntax)
  • Restart the salt-minion service

After doing this, if I do a deployment, the minion will show up in both masters with a Pending state. Let’s go ahead and deploy a server!

This is the result of the deployment

If I go to the SSC console, I will see this Minion pop up under the pending keys

And you can see that this will come up as pending on both masters! This means that the changes on the master.conf file on this instance of the deployment worked.

samba-0307 is pending on both masters

Now, if you accept the key on the SSC console, it will be accepted on both nodes. Accepting the key triggers a job that you can see on the Activity view.

Once the action is completed, the key will be accepted on both Masters, so both Masters wil be able to interact with the minion. This is a brief excerpt from the output of the job run

 {
    "return": {
      "minions": [
        "samba-0307"
      ]
    },
    "master_uuid": "fa821ecc-c2de-4c76-9477-0739835a5a63",
    "minion_id": "ssc-gool-master2_master",
    "jid": "20220404204133496265",
    "fun": "key.accept",
    "alter_time": "2022-04-04T20:41:50.005306",
    "full_ret": {
      "id": "ssc-gool-master2_master",
      "fun": "key.accept",

...
...
{
    "return": {
      "minions": [
        "samba-0307"
      ]
    },
    "master_uuid": "410143c8-f4e8-482a-9895-de0e8bd18537",
    "minion_id": "ssc-gool-master1_master",
    "jid": "20220404204133496265",
    "fun": "key.accept",
    "alter_time": "2022-04-04T20:41:57.999983",
    "full_ret": {
      "id": "ssc-gool-master1_master",
      "fun": "key.accept",

You can see that the action was executed on both masters, since they’re both part of a cluster.

25: Create a Job using the Salt State hosted in Github

The second to last step of this post. You will now create a job and use the Salt State located in the Github Repository.

  • Go to Config -> Jobs and click on Create Job
  • Under Command, select salt
  • Do not select any targets
  • Under Function select state.apply
  • Do not select an environment
  • Under states type in the name of the state, which is the name of the folder in Github. In this case, it will be apachenaming
  • Click on Save

Note: The SSC console does not have access to the GitFS filesystem, therefore, a State that exists in Github will never show up in the drop down list of States when the Job is being created. This is expected behavior, and why wou need to type the Salt state name.

26: Run the Job!

You made it! This is the last step! You will now test that you can actually run a job with a State file hosted on Git!

Go to ‘Minions’, select the minion that was deployed, click on Run Job and select the apachenaming job, and then click on ‘Run Now’

This job will install Apache on the server, and then configure the welcome page to show my name (LuchoDelorenzi) on it! This is the code of the state file (this is public on Github)

######################################################
# install apache webserver, start service, changePage
# 
###################################################### 
#install apache
apache-pkg:
  pkg:
    - installed
    - name: apache2
  service:
    - running 
    - name: apache2
    - require:
      - pkg: apache-pkg
      
#change page
change_page:
  cmd.run:
    - name: sed -i 's/Apache2 Ubuntu/LuchoDelorenzi/g' /var/www/html/index.html
    - require:
      - apache-pkg

And after the Job is completed, this is the result!

FINISHED!

If you made it this far, congratulations! I know this has been a really long post full with information, but I hope that you found it simple enough to consume and be able to attempt this deployment (and tweak things for your environment) based on this post!

Closing Note

As usual, looking forward to feedback in the comments. And if you liked the post, please share it! The more people I can help, the better!

Improving vRA 8 Custom Forms loading times – A practical example using vRO Configuration Elements as a Database!

Hello Everyone!

On today’s post, we will dive into a practical example to use vRealize Orchestrator Configuration Elements to help with a business need for vRA 8 Custom Forms!

The Problem

Customer X is using a single cloud template with multiple inputs backed by vRO Actions. The main input, and what defines pretty much all the rest, is the project selected. Given the complexity of the inputs, the cloud template can be used by all projects and by many different use cases.

Customer X was trying to improve the form loading times, which were around 10 seconds for the initial loading, plus 10 more seconds every time they changed the project in the form. This heavily impacted the user experience since it was giving a sensation of ‘slowness’ overall to anyone that was requesting the items.

The project defines, for example (there are more fields, but these are the ones we will use as example):

  • Hostname prefixes
  • Puppet roles
  • AD OUs
  • Portgroups
  • vCenter Folders

Each project has a ‘Project Map’ which contains different modifiers to then perform a different search in an external API, which has a cache of the objects needed, to reduce the time needed to gather the data (for example, sending API calls to vCenter to get folders)

However, the fact that the Project Map does not have all the information and needs to be processed in real time ends up adding more loading time to the form than desired.

A solution: vRO Configuration Elements

Emphasis on ‘A solution’ and not ‘THE solution’ since there could be other (even better!) ways to solve this problem, but this is how I approached it and will show it in this blog post.

vRO configuration elements, are originally used for example, for sets of variables that will be used in multiple internal actions/workflows, to avoid having the same data in many places, and for ease of managing. The configuration elements can be referenced in workflows or actions and the information is only changed in a single place.

However, there is another use we can give to configuration elements and that is using them as a Database!

All the configuration elements reside in the vRO DB, and the elements used can be of any of the types that exist within vRO.

For more information about configuration elements you can visit: https://docs.vmware.com/en/vRealize-Orchestrator/8.6/com.vmware.vrealize.orchestrator-using-client-guide.doc/GUID-F2F37F70-9F55-4D87-A3BB-F40B6D399FF8.html

So what is the approach here?

  • Creating a Configuration Elements category called ‘Projects’
  • Create one configuration element per project, within that category. The easiest way to accomplish this is to create one configuration element, define all the needed attributes, and then just duplicate that configuration element to match all the projects you need – In this case, since we need to retun this to vRA Custom Forms, mostly in drop-down form, we will be using string arrays
One configuration element per project, with the variables mentioned
  • An action that will return the values to the custom forms, using two inputs, the Project we want to get the information from, and the value that we want to get. That makes the action reusable by multiple fields in the form: In this case, I called it getConfigurationElementValue and it can be seen on the following link: https://github.com/luchodelorenzi/scripts/blob/master/getConfigurationElementValue.js
  • An action or workflow that will:
    • Get the data from the external API
    • Populate the configuration elements with that data

For this example, since I don’t have any external API in my lab, I will use static arrays to demonstrate the point in the code: The action is called updateConfigurationElements and can be seen in the following link https://github.com/luchodelorenzi/scripts/blob/master/updateConfigurationElements.js

This action/workflow can be scheduled to run every minute, every 5 minutes, depending on the need.

The data will be persisted in the vRO DB so that’s why I’m calling this a ‘database’ instead of a cache, however, it could very well be called a ‘persistent cache’ since all it is doing is to make the data available to the user as fast as possible but not doing any processing.

This workflow runs every 5 minutes and updates the values on all the existing projects (Configuration Elements)

The important thing to note here is that there isn’t any processing from the Custom Form to the vRO configuration elements when the user requests a catalog item!. Getting the data directly from the vRO DB without any processing at request time is what is going to give us the fastest loading times. All the processing is done in the background, without none of the requesters noticing!

  • The last step is to refer to the getConfigurationElementValue action in our custom form
    • A small caveat – the way vRA 8 and the ‘Project’ field works is that even though the project shows the user the names to be chosen, it is actually processing the IDs, so in this case I added a hidden field called ProjectName which is what I will be actually using to convert the IDs to names (since the configuration elements are based on the name)
Mapping the Project IDs to names
Using the getConfigurationElementValue action to get the values needed in the form

This is a small demo of how this works, take a look at the loading times for the form and changing the project! (And this is on a nested lab!)

Video Demo

Summary

To reiterate, the important things are:

  • No (or as little processing as possible if there is a field that cannot be used with configuration elements) should be done in the actions that are returning the data to the custom form
  • All the data should be processed in the background – The requester won’t be aware of it
  • Adding new projects it is as simple as duplicating one of the existing ones and changing the name. The way the workflows and actions are coded in this example will always look for every project (configuration element) below the ‘Projects’ folder
  • Getting the data out of the vRO DB directly via configuration elements instead of going to external sources, is the fastest way to get the values in the form.

Closing Note

I hope you found this interesting! It is using configuration elements in a way that might not be the most common usage, but it can bring great benefits to user experience when interacting with vRA requests. Having the data processed in the background and having really short form loading times will give the sensation of having more ‘speed’ to the tool itself!

Feel free to share this or leave a comment if you thought it was interesting!

Until next time!

Updating an Onboarded Deployment in vRA 8.x

Hello Everyone!

On today’s post, we will go through the process of updating an onboarded deployment in vRA 8.x

The onboarding feature allows customers to add VMs that were not deployed from vRA, to the vRA infrastructure. This means that these VMs are added to one or more deployments, and once they exist within vRA, operations such as power cycling, opening a remote console, or resizing CPU/RAM are now available.

However, there are scenarios in which customers would want to expand these deployments, not with new onboarded VMs, but with newly deployed VMs (or other resources) from vRA! These deployments will use an image, a flavor, could use a multitude of inputs, tagging, networks, etc. So how do we do this?

Onboarding the VMs using an auto-generated Cloud Assembly Template

The first thing we need to do, is to create an onboarding plan, select a name for our deployment, and select the VMs we’re going to onboard initially.

Creating the Onboarding Plan
Adding two VMs to be onboarded

On the deployments tab, we can rename the deployment if needed, but the most important part is to select Cloud Template Configuration and change it to Create Cloud Template in Cloud Assembly Format this will allow us to have a source for our deployment, that we can edit afterwards to allow for future growth

Cloud Template in Cloud Assembly format

It is important to note that the imageRef has no image available. Since this is not a vRA Deployment but an Onboarding, none of the resources are being deployed from any of the images. We will come back to this item later.

After saving this configuration and clicking on Run, our deployment will be onboarded

Updating the onboarded deployment to add a new VM in a specific network

If we check on the onboarded deployment, we will see that it is mapped to a specific Cloud Template (the one that was auto-generated earlier by the Onboarding Plan)

So if we were to do an update on this deployment, we need to edit that Cloud Template

I will now add a vSphere Machine resource as well as a vSphere Network:

inputs: {}
resources:
  Cloud_vSphere_Machine_1:
    type: Cloud.vSphere.Machine
    properties:
      image: 'ubuntu'
      cpuCount: 1
      totalMemoryMB: 1024
      networks:
        - network: '${resource.Cloud_vSphere_Network_1.id}'
  Cloud_vSphere_Network_1:
    type: Cloud.vSphere.Network
    properties:
      networkType: existing
      constraints: 
        - tag: env:vsphere  
  DevTools-02a:
    type: Cloud.vSphere.Machine
    properties:
      imageRef: no_image_available
      cpuCount: 1
      totalMemoryMB: 4096
  DevTools-01a:
    type: Cloud.vSphere.Machine
    properties:
      imageRef: no_image_available
      cpuCount: 1
      totalMemoryMB: 4096
  

This is what our template looks like now. So the next thing we should do is click on Update, right?

Update is Greyed out!

The update task is greyed out because ir Cloud Template does not have inputs. Since we don’t have inputs, what we need to do is to go to the Cloud Template, and instead of selecting Create a New Deployment we should select Update an Existing Deployment and then click on the onboarded deployment.

Updating the Onboarded Deployment

After clicking on Next, the plan is presented.

Notice something wrong here?

The update operation will attempt to re-create the onboarded VMs! That’s not something we want, and also, in this scenario, it will fail since there is no image mapping to deploy from!

What we want is to leave all the VMs that were previously onboarded, untouched, and only add our new VM and network. So how do we achieve this?

This is achieved by adding the ignorechanges parameter with a value of true to every resource in the cloud template that was previously onboarded – In this scenario, this would be our 2 DevTools VMs

Adding the ignoreChanges parameter

If we re-try updating the deployment now, the only tasks that should appear will be the ones for the new resources (VM and Network)

Update deployment showing the new tasks

After clicking on ‘deploy’ and waiting for it to finish, our deployment will now like this

Deployment updated with our new VM and network! Hooray!

Offboarding/Unregistering limitations

It is important to note that vRA’s limitations for unregistering VMs are still present. The only VMs that can be unregistered from vRA are the ones that were previously onboarded. VMs that were deployed from vRA will not be able to be unregistered without deletion. The fact that the deployment VMs are part of an Onboarded Deployment does not change this.

Closing Note

I hope you enjoyed this post! When I started working on this use case I figured it was not as trivial as I thought, and after doing research and testing, found this walkthrough/solution.

Let me know if this was useful in the comments!

Until next time!