SaltStack Config Enterprise Install, Multi-Master setup and Git repo configuration!

Featured

Hello Everyone! On today’s post, I’m going to do a step by step walkthrough of a SaltStack Config (from now on, SSC) enterprise install. The SSC enterprise install is meant to be used in production-grade environments, and can handle up to 5000 Salt Minions per master!

I will also cover how scale-out the deployment and add a new Salt Master (in a cluster configuration) to the deployment after it is finished.

In addition to this, I will cover how to configure a Git repo as a shared storage across masters.

Lastly, I will cover how to prepare a vRA Cloud Template to install the Salt Minion, configure it to use multiple masters, and run a Salt State on the deployment!

Let’s start! Buckle up!

Architecture

This deployment is going to have:

  • One VM for PostgreSQL and Redis – Required for persistent & in-memory database components. These two components could be also separated across two different VMs (and configured in HA, but only for manual failover).
  • One VM for RaaS (Responder as a Service) – This is going to be the GUI of SSC. The RaaS component can also be deployed in a cluster mode using an external load balancer, but I’m going to use a single one in this post.
  • Two VMs, one for each Salt Master, that will form a cluster. The secondary master and the cluster will be generated after the initial deployment.
  • Both Salt Masters will be configured to use a Github repository.

All VMs are running Centos 7 as the operating system.

Architecture

This is what my servers look like, from a vSphere point of view.

VMs

List of Steps

What do you need to do to carry out this deployment?

The first list of steps is based on this LINK, which is the ‘Installing and Configuring Saltstack Config’ Official Documentation from VMware.

  • Prepare the template for the VMs (you can prepare each VM separately but there are common items across all VMs)
  • Deploy the 4 VMs (take note of the IP addresses)
  • Prepare the VM that will become the Primary Salt Master
  • Prepare the VMs that will only be Salt Minions (the PostgreSQL / Redis & the RaaS)
  • Download & Copy the SSC installer to the Primary Master
  • Copy and edit the top state files
  • Edit the SSC settings Pillar file
  • Apply the highstates to the nodes
  • Install the License Key
  • Install and configure the Master Plugin on the Primary Salt Master
  • Log in for the first time and change the default credentials
  • Accept the Primary Salt Master key
  • Optional: Configure Backup for files (if not using a complete backup solution), Set up custom SSL certificates, SSO. (won’t be doing this as a part of this post) -> Link1, Link2, Link3

At this point, you’re going to have a functional SSC enterprise install, but you will only have a single master node. You still need to configure the Secondary Master, the cluster, and the repository!

The second list of steps is based on multiple sources and testing, since there isn’t a single source of information to configure all of this. Which is why I’m attempting to write this and condense it! I will be adding links throughout the steps.

  • Prepare the VM that will become the Secondary Salt Master
  • Copy the SSC installer to the Secondary Master
  • Prepare all the other minions to use Primary and Secondary Salt Masters
  • Copy Primary master key to the Secondary Master
  • Edit the RaaS configuration file (master plugin) on the primary master to add the Cluster ID
  • Install and configure the Master Plugin in the Secondary Master
  • Edit the RaaS configuration file (master plugin) on the Secondary Master to add the Cluster ID
  • Start Salt-Master on the Secondary Master and accept the Secondary Salt Master key
  • Install GitPython on both Salt Masters
  • Configure the GitFS filesystem and Github repository
  • Configure a Cloud Template on vRA to install the Salt Minion and configure the two masters
  • Create a Job using the Salt State hosted in Github
  • Run the Job!

Don’t be scared! I will be explaining every single step so you’re also able to have a successful deployment. Let’s start!

1: Prepare the template for VMs

You need to install OpenSSL, EPEL, and two libraries for Python (cryptography and OpenSSL) so this is what you need to run:

sudo yum install openssl -y
sudo yum install epel-release -y
sudo yum install python36-cryptography -y 
sudo yum install python36-pyOpenSSL -y 

You can also, disable firewalld (if you have any other firewalling solution) to avoid headaches with the inter-component communication.

sudo systemctl stop firewalld
sudo systemctl disable firewalld

After this is done, you can shutdown this template and use it to clone the 4 VMs needed in the deployment.

2: Deploy the 4 VMs

Self explanatory, deploy the VMs in your vSphere Environment.

3: Prepare the Primary Salt Master VM

You need to configure the Salt Repository, install the Salt Master and the Salt Minion.

sudo yum install https://repo.saltstack.com/py3/redhat/salt-py3-repo-latest.el7.noarch.rpm -y 

sudo yum clean expire-cache -y 

sudo yum install salt-master -y
sudo yum install salt-minion -y

Create a master.conf file in the /etc/salt/minion.d directory and add the following text to configure the minion to use itself as a Master.

master: localhost

Create a the minion_id file in the /etc/salt/ with a descriptive name for the minion using vi, in this scenario, for example, my Primary Master’s minion_id is ssc-gool-master1. This will be autogenerated based on hostname on the first run of the salt-minion service if its not set previously.

[centos@ssc-gool-master1 salt]$ cat minion_id
ssc-gool-master1

Enable and start the services

sudo systemctl start salt-master
sudo systemctl enable salt-minion
sudo systemctl start salt-minion

4: Prepare the PostgreSQL/Redis & RaaS VMs

You need to configure the Salt Repository, and Install the Salt Minion.

sudo yum install https://repo.saltstack.com/py3/redhat/salt-py3-repo-latest.el7.noarch.rpm -y 

sudo yum clean expire-cache

sudo yum install salt-minion -y 

Create a master.conf file in the /etc/salt/minion.d directory and add the following text to configure the minion to use the Primary Salt Master as its Master

master: IP_OF_MASTER

Set the minion_id (located in /etc/salt) to a descriptive name using vi, in this scenario, for example, my RaaS minion_id is ssc-gool-raas.

[centos@ssc-gool-raas salt]$ cat minion_id
ssc-gool-raas

My PostgreSQL + Redis minion_id is ssc-gool-psqlr

[centos@ssc-gool-psqlr ~]$ cat /etc/salt/minion_id
ssc-gool-psqlr

Enable and start the salt-minion service

sudo systemctl enable salt-minion
sudo systemctl start salt-minion

5: Download & Copy the SSC installer to the Primary Master

Download the SSC installer from Customer Connect (https://customerconnect.vmware.com/)

Copy the installer to a folder within the Primary Master (it can be the root directory as well), for example, /ssc-installer, and assuming our file is called ssc_installer.tar.gz.

scp ssc_installer.tar.gz USERNAME@IP_ADDRESS://ssc-installer/

Then extract the installer to a folder

tar -xzvf ssc_installer.tar.gz

6: Copy and edit the top state files

The top state files will be used by the orchestration to install the RaaS, Redis and PostgreSQL nodes.

At this point, you should take note of the Minion ID and the IP addresses of your three nodes, since you will be using them in the following steps. In my case, this is the information:

MINION ID: IP ADDRESS
ssc-gool-master1: 10.0.0.2
ssc-gool-psqlr: 10.0.0.3
ssc-gool-raas: 10.0.0.4

Now, you need to copy and edit the orchestration configuration files.

Important: The instructions below assume that this is a ‘greenfield’ Salt installation. If this is not the case, you might need to edit the following commands to work within your directory/folder structure.

Navigate into the sse-installer folder (this is the folder that was extracted from the tar.gz file) and run the following commands:

sudo mkdir /srv/salt
sudo cp -r salt/sse /srv/salt/
sudo mkdir /srv/pillar
sudo cp -r pillar/sse /srv/pillar/
sudo cp -r pillar/top.sls /srv/pillar/
sudo cp -r salt/top.sls /srv/salt/

In the /srv/pillar directory, you now have a file named top.sls. Edit this file to define the list of Minion IDs (not IP addresses) that you recorded previously. This is how it looks in my environment

{# Pillar Top File #}

{# Define SSE Servers #}
{% load_yaml as sse_servers %}
  - ssc-gool-master1
  - ssc-gool-psqlr
  - ssc-gool-raas
{% endload %}

Since as mentioned earlier, my 3 Minion IDs are ssc-gool-master1, ssc-gool-psqlr and ssc-gool-raas.

Also make sure that in the /srv/salt directory you also have a file named top.sls that looks like this:

base:

  {# Target SSE Servers, according to Pillar data #}
  # SSE PostgreSQL Server
  'I@sse_pg_server:{{ grains.id }}':
    - sse.eapi_database

  # SSE Redis Server
  'I@sse_redis_server:{{ grains.id }}':
    - sse.eapi_cache

  # SSE eAPI Servers
  'I@sse_eapi_servers:{{ grains.id }}':
    - sse.eapi_service

  # SSE Salt Masters
  'I@sse_salt_masters:{{ grains.id }}':
    - sse.eapi_plugin

7: Edit the SSC settings pillar file

You need to edit four different sections in the SSC settings pillar file to provide the values that are appropriate for the environment. These settings will be used by the configuration state files to deploy and manage your SSC deployment.

Navigate to the /srv/pillar/sse directory and edit the sse_settings.yaml file.

Section #1: Change the values of the four variables to match your Minion IDs. In my case, this looks like this:

# Section 1: Define servers in the SSE deployment by minion id
servers:

  # PostgreSQL Server (Single value)
  pg_server: ssc-gool-psqlr

  # Redis Server (Single value)
  redis_server: ssc-gool-psqlr

  # SaltStack Enterprise Servers (List one or more)
  eapi_servers:
    - ssc-gool-raas

  # Salt Masters (List one or more)
  salt_masters:
    - ssc-gool-master1

Section #2: Edit the following variables

  • pg_endpoint: use the IP address (or DNS name) of the PostgreSQL server. In my environment, this is 10.0.0.3.
  • pg_port: Port for PostgreSQL. In my environment, I left the default values
  • pg_username and pg_password: Credentials for the user that RaaS will use to authenticate to PostgreSQL

This section looks like this:

# Section 2: Define PostgreSQL settings
pg:

  # Set the PostgreSQL endpoint and port
  # (defines how SaltStack Enterprise services will connect to PostgreSQL)
  pg_endpoint: 10.0.0.3
  pg_port: 5432

  # Set the PostgreSQL Username and Password for SSE
  pg_username: salteapi
  pg_password: VMware1

Section #3: Repeat the previous steps but this time, to match your Redis parameters. Since we’re using the same server for both PostgreSQL and Redis, the IP will be the same.

# Section 3: Define Redis settings
redis:

  # Set the Redis endpoint and port
  # (defines how SaltStack Enterprise services will connect to Redis)
  redis_endpoint: 10.0.0.3
  redis_port: 6379

  # Set the Redis Username and Password for SSE
  redis_username: saltredis
  redis_password: VMware1

Section #4: Edit the variables that are related to the RaaS node

  • Since this is a fresh installation, do not change the eapi_username and eapi_password values. You will change the default password at a later step
  • eapi_endpoint: set it to match the IP address of your RaaS node. In my environment, this is 10.0.0.4
  • eapi_ssl_enabled: default is set to true. SSL validation is not required by the installer but it will be likely a security requirement in environments that use CA certificates.
  • eapi_ssl_validation: default is set to false. This means that the installer will not validate the SSL certificate.
  • eapi_standalone: default is set to false. This variable would be true in the case of the LCM install, in which all components are shared in a single node.
  • eapi_failover_master: default is set to false. This would be used if you were to configure a Multi Master configuration in failover mode (not active-active) and from within the installer. This will keep its default value since the scaling out will be done afterwards.
  • cluster_id: This variable defines the ID for a set of Salt masters when configured in a multi-master configuration. The default value should be left here, this will be edited at a later step, once the deployment is already running.

This is what my file looks like:

# Section 4: eAPI Server settings
eapi:

  # Set the credentials for the SaltStack Enterprise service
  # - The default for the username is "root"
  #   and the default for the password is "salt"
  # - You will want to change this after a successful deployment
  eapi_username: root
  eapi_password: salt

  # Set the endpoint for the SaltStack Enterprise service
  eapi_endpoint: 10.0.0.4

  # Set if SaltStack Enterprise will use SSL encrypted communicaiton (HTTPS)
  eapi_ssl_enabled: True

  # Set if SaltStack Enterprise will use SSL validation (verified certificate)
  eapi_ssl_validation: False

  # Set if SaltStack Enterprise (PostgreSQL, eAPI Servers, and Salt Masters)
  # will all be deployed on a single "standalone" host
  eapi_standalone: False

  # Set if SaltStack Enterprise will regard multiple masters as "active" or "failover"
  # - No impact to a single master configuration
  # - "active" (set below as False) means that all minions connect to each master (recommended)
  # - "failover" (set below as True) means that each minion connects to one master at a time
  eapi_failover_master: False

There is also a Section #5, but none of the values need to be edited at this step. These are the customer_id variable, which is a variable that uniquely identifies a SSC deployment, and the cluster_id variable, which will be edited once the deployment is already running and the scale-out is done.

8: Apply the highstates to the nodes

At this point, it would be wise to take snapshots of all your nodes, in case something goes wrong with applying the highstates, instead of having to troubleshoot a failed installation, it might be easier to rollback to the snapshot state and start over from this point.

Having said that, to apply the highstates, you need to do the following:

  • Accept the keys on your Primary Master, you can do that by running the command sudo salt-key -A which will accept all unaccepted keys (at this point, 3)
  • On your Salt Master, sync your grains to confirm that the Salt Master has the grain data needed for each minion. Since this is a fresh install, you can just run the command to target all the minions, which at this point are just 3.
sudo salt \* saltutil.refresh_grains
  • Then, run the following command to refresh the pillar data on all the minions
sudo salt \* saltutil.refresh_pillar
  • Lastly, run the following command to confirm the return data for your pillar is correct
sudo salt \* pillar.items

Confirm that the minions have received the pillar data that you edited on the sse_settings.yaml file, such as IP addresses, Minion IDs, etc

Now that you confirmed the data, it is time to apply the highstates to each node, by running the following command: sudo salt MINION_ID state.highstate – The PostgreSQL database should always be applied first

Which in my environment would look like:

sudo salt ssc-gool-psqlr state.highstate
sudo salt ssc-gool-raas state.highstate
sudo salt ssc-gool-master1 state.highstate

Confirm that the result of applying the highstates is succesful.

Note: you might get a ‘Authentication Error Occurred’ when applying the highstate to the Salt Master. This is expected, and it is deplayed because the Salt Master has not authenticated to the RaaS node yet. This will be solved at a later step.

If this has been successful, you now have a functioning install of SSC. But you still need steps to complete, let’s continue!

9: Install the License Key

To install the License key

  • Get your License Key from My VMware / Customer Connect (a vRA license is used)
  • Create a file with a filename ending in _license such as ssc_license for example
  • Edit the file and add your license key number.
  • Change ownership of the license file and copy the file to the /etc/raas directory
chown raas:raas ssc_license
mv raas.license /etc/raas
  • Restart the RaaS service: sudo systemctl restart raas

10: Install and configure the Salt Master Plugin

The Salt Master plugin allows the Salt Masters to communicate with SSC. The master plugin is installed on every Salt Master in your environment that communicates with SSC. At this step, you will only install it on the primary Salt Master

  • Log in to your Salt Master
  • The master plugin is located in the sse-installer/salt/sse/eapi_plugin/files directory. cd into that directory.
  • Install the Master Plugin by manually installing the Python wheel, using the following command, and replacing the exact name of the wheel file.
sudo pip3 install SSEAPE-file-name.whl --prefix /usr
  • Verify that the /etc/salt/master.d directory exists
  • Generate the master configuration settings
sudo sseapi-config --all > /etc/salt/master.d/raas.conf

Note: I had to do this step while being logged in as the root user since I was not able to generate the file even with a sudoer user. If this is your case, just switching to the root user and running the command will do the trick.

  • Edit the generated raas.conf file and update it to use your RaaS server
    • sseapi_server: Since you enabled SSL at a previous step, the URL should be https://IP_ADDRESS_OF_RAAS – in my environment, it would be 10.0.0.4
    • sseapi_ssl_validate_cert: However, since you’re not using a CA-signed cert, you should disable the validation to allow for the communication between the Master Plugin and RaaS

This file has more parameters that can be edited at this stage, for example, to set a custom certificate, or specific performance configurations. For more information, visit: LINK

  • Restart the Master Service: sudo systemctl restart salt-master

You can also check and edit the RaaS configuration file to edit RaaS related parameters. I won’t be covering them in this post and will be using the default values, but more information can be found at: LINK

11: Log in and change the default credentials

Log in to the SSC interface with the default credentials

Then go to Administration -> Authentication -> Local Users, and change the password for the root user

12: Accept the Salt Master Key

Go to Administration -> Master keys. You will see your Master node with a Key in the ‘pending’ state. Accept it.

At this point, you should see your minions pop up in the ‘Minions’ screen.

You can then run a simple command, such as test.ping to make sure that you can connect to your minions. For example:

Testing ping command from salt.master

You can also test this from the RaaS console, by selecting the minions and running the same job

Running test.ping job from the console

Congrats, if you made it here, you have a functioning distributed install of SSC! Now you will scale this out to allow for multiple masters and a shared repository!

A little break

At this point, we have configured our initial deployment. From now on, I will cover how to scale this out to add a secondary master, and configure a git repository!

14: Prepare the VM that will become the Secondary Salt Master

Follow Step #3 from the list, but this time using the Secondary Master VM!

15: Copy the SSC installer to the Secondary Master

Follow Step #5 from the list, but only copy the file, since you already downloaded it in that step.

16: Prepare all minions to use both masters

At this step, you need to edit the /etc/salt/minion.d in all the minions to use both masters. In the case of the nodes that are masters, you can just keep the localhost value and add the value of the secondary master to each node. In the case of the nodes that are minions (PostgreSQL/Redis, RaaS) you can append the IP of the secondary master to the file. Keep in mind that this will now become a list, and the syntax changes. The files should look like this (using the IPs from my environment)

  • For Primary Master:
[root@ssc-gool-master1 centos]# cat /etc/salt/minion.d/master.conf
master:
  - localhost
  - 10.0.0.5
  • For Secondary Master:
[centos@ssc-gool-master2 pki]$ cat /etc/salt/minion.d/master.conf
master:
  - localhost
  - 10.0.0.2
  • For the PostgreSQL/Redis & RaaS nodes:
[centos@ssc-gool-raas /]$ cat /etc/salt/minion.d/master.conf
master:
  - 10.0.0.2
  - 10.0.0.5

Restart the Salt Minion service on each node after editing the files: sudo systemctl restart salt-minion

17: Copy the Primary Master Key to the secondary master node

This is a requirement to be able to use a redundant master (regardless of it being configured in an active-active configuration or active-passive). The masters need to share the private and public key.

You should log in to the Primary Master and run the following commands to copy the files. This can be done with the root user if you run into any issue with accessing the folder. Overwrite the existing files if prompted.

cd /etc/salt/pki/master
scp master.pem USERNAME@IP_OF_SECONDARY_MASTER:/etc/salt/pki/master/
scp master.pub USERNAME@IP_OF_SECONDARY_MASTER:/etc/salt/pki/master/

Then, log in to the Secondary Master and restart the Master Service: sudo systemctl restart salt-master

18: Edit the RaaS configuration file (Master Plugin) on the primary master to add the Cluster ID

Since we’re going to place both masters in the same cluster, we need to make RaaS aware of this. This configuration is handled in the Master Plugin configuration.

To do this change, open the /etc/salt/master.d/raas.conf file on the Primary Master and edit the value of the sseapi_cluster_id variable. In my environment, this looks like this:

sseapi_cluster_id: goolcluster                                         # SSE cluster ID for this master (optional)

As we saw on the Architecture image, my cluster will be called goolcluster

Then, restart the Salt Master service on the Primary Master: sudo systemctl restart salt-master

19: Install and configure the Master Plugin in the Secondary Master

Follow Step #10 using the Secondary Master VM.

20: Edit the RaaS configuration file (Master Plugin) on the secondary master to add the Cluster ID

Follow Step #18 using the Secondary Master. Make sure to use the same Cluster ID, in this case, goolcluster.

21: Start Salt-Master on the secondary master and accept the second Salt Master Key

Log in to the Secondary Master, then run the following command to start the Salt Master service: sudo systemctl start salt-master

Then, log in to SSC using a browser, and accept the Master Key for the secondary master. Once it is accepted, it should look like this:

Both master keys accepted

At this point, since all minions are configured to use both masters, you will get pending minion keys in the SSC console. You can accept them there.

Note: Since both masters are part of the same cluster, SSC will recognize them as the same node. They will show up in the ‘Pending’ view of keys, but they will show up as already accepted. You need to accept them again (since this will be accepting the key in the secondary master). In the future, new minions will only need to be accepted once, since accepting on SSC will run the job to accept keys on both Salt Masters

The next steps will be about configuring the GitFS filesystem, which is one of the ways you can use a shared filesystem for both masters. It is a requirement to have a shared filesystem across masters, otherwise you could have inconsistent information depending on what master is being used for any given Job instance.

22: Install GitPython on both Salt Masters

Configuring a GitFS filesystem in a Salt Master can be accomplished through two methods:

  • Via GitPython
  • Via pygit2

I was having trouble with pygit2 in CentOS and getting Salt to recognize the version (this has been reported in multiple Git and Stackoverflow posts), so I ended up using GitPython instead, and this is what I will be describing.

To install GitPython, and its dependencies (such as the git cli), log in to both Salt Masters and run the following command: sudo pip3 install GitPython

After installing it, make sure that it shows up as being used by Salt. Salt uses its own Python version so some packages don’t always get recognized.

Run the following command: salt -V

[centos@ssc-gool-master2 ~]$ salt -V
Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.9.1
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: 4.0.9
     gitpython: 3.1.20
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: 3.14.1
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: 5.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

Salt Extensions:
        SSEAPE: 8.6.2.11

You can see that GitPython shows up as installed and with version 3.1.20, while pygit2 is not installed.

23: Configure the GitFS filesystem and Github Repository

The URL for the repository that I’m using now is public, and it is https://github.com/luchodelorenzi/saltstack -> you can use it as well for doing the same tests I will be doing on this deployment.

All the steps of this configuration need to be done on both Salt Masters, since they’re now running in a cluster

  • Edit the fileserver_backend parameter and add the ‘gitfs’ filesystem to the /etc/salt/master.d/raas.conf file.
# Enable SSE fileserver backend
fileserver_backend:
  - sseapi
  - roots
  - gitfs
  • Create a new file in the same directory, called gitfs.conf and add the following parameters
gitfs_provider: gitpython

gitfs_update_interval: 60

gitfs_base: main

gitfs_remotes:
 - https://github.com/luchodelorenzi/saltstack.git

Note: the same raas.conf file could have been used to append the gitfs parameters. However, Salt will look for all *.conf files in the master.d directory, so separating this in a different file could make it easier to maintain / check.

What do each of the parameters mean?

  • gitfs_provider: The provider that will be used to leverage GitFS. In this case, GitPython
  • gitfs_update_interval: Update interval for gitfs remotes.
  • gitfs_base: Defines what branch or tag is used as the base environment. The main branch on my repository is main, but it can change depending on yours
  • gitfs_remotes: List of the repositories. Only adding one in this deployment. You can have multiple deployments, and some parameters are overridable per remote.

There are multiple other parameters for GitFS. For more information please follow this LINK on the GitFS section. You can also follow this other LINK for a GitFS walkthrough.

After configuring this, restart the Salt Master on both nodes, by running the following command: sudo systemctl restart salt-master

Now, you need to check if our files are being read from Github! Since the mapping was done to the base environment, running the following command will show every state file in that environment. For example:

[centos@ssc-gool-master2 master.d]$ sudo salt-run fileserver.file_list saltenv=base
- _beacons/status.py
- apachenaming/init.sls
- presence/init.sls

The apachenaming state resides on Github, as you can see here:

Github Repository

So hooray, we can now use this Salt state from Github, and this is shared across both masters, has version control, you can use multiple repos, multiple branches. Pretty cool, isn’t it?

24: Configure a Cloud Template to Install the Salt Minion and Configure Two Masters

Since the vRA integration with SSC does not allow to integrate with Multiple Masters, I will use CloudConfig to perform the initial installation and configuration. As a prerequisite for this, your template should be prepared to use CloudConfig

This is the CloudConfig Code in the Template I’m using:

cloudConfig: |
        #cloud-config
        hostname: ${self.resourceName}
        runcmd:
          - curl -L https://bootstrap.saltstack.com -o install_salt.sh
          - sudo sh install_salt.sh -A 10.0.0.2
          - sudo chown ubuntu /etc/salt/minion.d
          - sudo rm /etc/salt/minion.d/99-master-address.conf
          - sudo echo -en 'master:\n  - 10.0.0.2\n  - 10.0.0.5' > /etc/salt/minion.d/master.conf
          - sudo systemctl restart salt-minion

What am I doing here?

  • Download the install_salt.sh bootstrap script
  • Run it to install the Salt Minion and point it to one of the masters
  • Change ownership of the minion.d directory to the ubuntu user which is the user that’s being used by CloudConfig in this template
  • Remove the 99-master-address.conf that was generated during the Salt Minion install
  • Create a new file with the addresses of the two Salt Masters (and using the correct list syntax)
  • Restart the salt-minion service

After doing this, if I do a deployment, the minion will show up in both masters with a Pending state. Let’s go ahead and deploy a server!

This is the result of the deployment

If I go to the SSC console, I will see this Minion pop up under the pending keys

And you can see that this will come up as pending on both masters! This means that the changes on the master.conf file on this instance of the deployment worked.

samba-0307 is pending on both masters

Now, if you accept the key on the SSC console, it will be accepted on both nodes. Accepting the key triggers a job that you can see on the Activity view.

Once the action is completed, the key will be accepted on both Masters, so both Masters wil be able to interact with the minion. This is a brief excerpt from the output of the job run

 {
    "return": {
      "minions": [
        "samba-0307"
      ]
    },
    "master_uuid": "fa821ecc-c2de-4c76-9477-0739835a5a63",
    "minion_id": "ssc-gool-master2_master",
    "jid": "20220404204133496265",
    "fun": "key.accept",
    "alter_time": "2022-04-04T20:41:50.005306",
    "full_ret": {
      "id": "ssc-gool-master2_master",
      "fun": "key.accept",

...
...
{
    "return": {
      "minions": [
        "samba-0307"
      ]
    },
    "master_uuid": "410143c8-f4e8-482a-9895-de0e8bd18537",
    "minion_id": "ssc-gool-master1_master",
    "jid": "20220404204133496265",
    "fun": "key.accept",
    "alter_time": "2022-04-04T20:41:57.999983",
    "full_ret": {
      "id": "ssc-gool-master1_master",
      "fun": "key.accept",

You can see that the action was executed on both masters, since they’re both part of a cluster.

25: Create a Job using the Salt State hosted in Github

The second to last step of this post. You will now create a job and use the Salt State located in the Github Repository.

  • Go to Config -> Jobs and click on Create Job
  • Under Command, select salt
  • Do not select any targets
  • Under Function select state.apply
  • Do not select an environment
  • Under states type in the name of the state, which is the name of the folder in Github. In this case, it will be apachenaming
  • Click on Save

Note: The SSC console does not have access to the GitFS filesystem, therefore, a State that exists in Github will never show up in the drop down list of States when the Job is being created. This is expected behavior, and why wou need to type the Salt state name.

26: Run the Job!

You made it! This is the last step! You will now test that you can actually run a job with a State file hosted on Git!

Go to ‘Minions’, select the minion that was deployed, click on Run Job and select the apachenaming job, and then click on ‘Run Now’

This job will install Apache on the server, and then configure the welcome page to show my name (LuchoDelorenzi) on it! This is the code of the state file (this is public on Github)

######################################################
# install apache webserver, start service, changePage
# 
###################################################### 
#install apache
apache-pkg:
  pkg:
    - installed
    - name: apache2
  service:
    - running 
    - name: apache2
    - require:
      - pkg: apache-pkg
      
#change page
change_page:
  cmd.run:
    - name: sed -i 's/Apache2 Ubuntu/LuchoDelorenzi/g' /var/www/html/index.html
    - require:
      - apache-pkg

And after the Job is completed, this is the result!

FINISHED!

If you made it this far, congratulations! I know this has been a really long post full with information, but I hope that you found it simple enough to consume and be able to attempt this deployment (and tweak things for your environment) based on this post!

Closing Note

As usual, looking forward to feedback in the comments. And if you liked the post, please share it! The more people I can help, the better!

Deploying from a Master Template using multiple service broker forms, using vRA API

Hello Everyone! I hope you’re having a good end of the year.

On today’s post, I will talk about a specific use case and one of the ways to solve it:

The business need was to do the following:

  • Have a single master template with dozens of inputs, to fullfill every deployment need.
  • This master template will be consumed by multiple projects and different users
  • Availability of inputs need to change based on the consumer / project
  • Visibility of inputs needs to change based on the consumer / project
  • Format of inputs needs to change based on the consumer / project
  • Source (in case of external data) for input data needs to change based on the consumer / project

You can see that we’re hitting a few limitations in the vRA OOB code:

  • The ‘conditional’ values for existence, visibility, format, etc, for a service broker form inputs don’t allow for this level of customization
  • A single cloud template can’t have more than one service broker form attached to it

So how do we solve this business need? Here comes the API solution!

Important: all the API information in swagger format (everything that was used in making this solution) is available in your vRA instance at https://VRA_FQDN/automation-ui/api-docs

API Documentation URL

What does the solution need to accomplish?

I will make a quick summary of what the code needs to do

  • Grab the inputs from a service broker form (and the requester ID, this is important)
  • Save the actual deployment name, but make the API deployment (mapped to the vRO workflow) use a temporary
  • Use the inputs to deploy the master template blueprint via the API (since we’re using the API and we don’t have access to the requester’s credentials, the API call will be executed by an administrative account configured in vRO)
  • Poll the master template blueprint deployment until it is successful
  • Once it is successful, change the owner to the original requester
  • Once that is done, destroy the temporary deployment that generates the API one.

Does that make sense?

I’ll break it down a bit:

These are the blueprint inputs:

inputs:
  instances:
    type: integer
    default: 1
  flavor:
    type: string
    default: SMALL
  image:
    type: string
    default: centos
  network:
    type: string
    default: 'network:web'
  environment:
    type: string
    default: 'env:vsphere'

These inputs (and a bit more) are the ones that are going to be used in the API call to the blueprint-request API -> part of that code. I create the body of the call with the inputs

var blueprintId = "01b5b4db-48b6-4b29-b062-a7dc1c5d9c93" // hardcoded master template
var blueprintInputs = {}
blueprintInputs.instances = instances
blueprintInputs.environment = environment
blueprintInputs.image = image
blueprintInputs.network = network
blueprintInputs.flavor = flavor
var blueprintBody = {}
blueprintBody.blueprintId = blueprintId
blueprintBody.blueprintVersion = "1"
blueprintBody.deploymentName = realDeploymentName
blueprintBody.projectId = projectId
blueprintBody.reason = "X"
blueprintBody.inputs = blueprintInputs
blueprintBodyString = JSON.stringify(blueprintBody)
System.log(blueprintBodyString)
var request = restHost.createRequest("POST", "/blueprint/api/blueprint-requests", blueprintBodyString);

Now those inputs need to be part of the action and the workflow we’re going to use, as you can see here:

vRO Action Inputs

You can see that we have some extra inputs that are not in the blueprint!

  • All the REST configuration will be variables of the workflow that is calling the action. This will use the previously configured REST host and the credentials
  • ProjectId is needed for the deployment, and we get that from the Service Broker Form
  • ownerId is needed for the deployment change owner action, and we get that from the execution context of the workflow that is being called by the service broker form
  • realDeploymentName is the actual deployment name (remember that the Service Broker form will use the temporary deployment name, and the actual deployment will use the name you input.

So what does the structure look like? In this case, we have two offerings for two projects:

Each of them have different configurations for the fields, for example:

projectId and temporary deployment name are visible fields, the rest is editable
projectId and temporary deployment name are not visible, plus, a bunch of fields aren’t editable

These two service broker forms map to two vRO workflows:

The two workflows

Why do I need two workflows? because as mentioned before, I can’t use two service broker forms for the same catalog item. However, these two vRO workflows are just wrappers of the ‘main’ workflow

The ‘offering’ workflow is just a wrapper for the base workflow

However, there is an important caveat here. The ownerId will only be part of the workflow that is called by Service Broker, in this case, the wrapper one. So the information for the requester is on the execution context of this workflow.

And I need to pass that to the base workflow. So how do I do that? By extracting the property from the execution context, and then passing it to the base workflow.

The only thing left now is to destroy the temporary deployment after it is done with the API one. So how do we do that?

We have a workflow that will delete a deployment via API – It basically calls the the deployments API with a DELETE action

//delete Deployment
System.log(deploymentId)
var request = restHost.createRequest("DELETE", "/deployment/api/deployments/"+deploymentId);
request.contentType = "application/json";
request.setHeader("accept", "application/json");
request.setHeader("Authorization", "Bearer " + tokenResponse)
 
//Attempt to execute the REST request
try {
    response = request.execute();
    jsonObject = JSON.parse(response.contentAsString);
    System.log(response.contentAsString)
}
catch (e) {
    throw "There was an error executing the REST call:" + e;
}

The deploymentId comes from the subscription run. Since this is run from the Event Broker state ‘Deployment Completed’, we have the deploymentId available there

deploymentId parameter on the event broker state

So we just extract it (the same way we did with the execution context) but from the inputProperties (payload used in the event broker subscriptions)

So how does all of this look in an actual run? Let’s show it in a video! (takes around 6 minutes, you will see the temporary deployment being generated, then the actual one via the API, the owner name change, and the destruction of the original one.

Deployment DEMO!!!

And I’m going to attach the code of the two most important actions that form the workflows (deployViaApi and deleteDeployment) here:

https://github.com/luchodelorenzi/scripts/blob/master/deleteViaAPI.js

https://github.com/luchodelorenzi/scripts/blob/master/deployViaAPI.js

Summary

I hope you enjoyed reading and understanding this as much as I did while trying to come up with this solution and then making this a reality. This idea can be heavily customized to suit other use cases (and grow this one way more) but the principles used here should still apply!

Please leave your feedback in the comments if you liked it, and share it!

Improving vRA 8 Custom Forms loading times – A practical example using vRO Configuration Elements as a Database!

Hello Everyone!

On today’s post, we will dive into a practical example to use vRealize Orchestrator Configuration Elements to help with a business need for vRA 8 Custom Forms!

The Problem

Customer X is using a single cloud template with multiple inputs backed by vRO Actions. The main input, and what defines pretty much all the rest, is the project selected. Given the complexity of the inputs, the cloud template can be used by all projects and by many different use cases.

Customer X was trying to improve the form loading times, which were around 10 seconds for the initial loading, plus 10 more seconds every time they changed the project in the form. This heavily impacted the user experience since it was giving a sensation of ‘slowness’ overall to anyone that was requesting the items.

The project defines, for example (there are more fields, but these are the ones we will use as example):

  • Hostname prefixes
  • Puppet roles
  • AD OUs
  • Portgroups
  • vCenter Folders

Each project has a ‘Project Map’ which contains different modifiers to then perform a different search in an external API, which has a cache of the objects needed, to reduce the time needed to gather the data (for example, sending API calls to vCenter to get folders)

However, the fact that the Project Map does not have all the information and needs to be processed in real time ends up adding more loading time to the form than desired.

A solution: vRO Configuration Elements

Emphasis on ‘A solution’ and not ‘THE solution’ since there could be other (even better!) ways to solve this problem, but this is how I approached it and will show it in this blog post.

vRO configuration elements, are originally used for example, for sets of variables that will be used in multiple internal actions/workflows, to avoid having the same data in many places, and for ease of managing. The configuration elements can be referenced in workflows or actions and the information is only changed in a single place.

However, there is another use we can give to configuration elements and that is using them as a Database!

All the configuration elements reside in the vRO DB, and the elements used can be of any of the types that exist within vRO.

For more information about configuration elements you can visit: https://docs.vmware.com/en/vRealize-Orchestrator/8.6/com.vmware.vrealize.orchestrator-using-client-guide.doc/GUID-F2F37F70-9F55-4D87-A3BB-F40B6D399FF8.html

So what is the approach here?

  • Creating a Configuration Elements category called ‘Projects’
  • Create one configuration element per project, within that category. The easiest way to accomplish this is to create one configuration element, define all the needed attributes, and then just duplicate that configuration element to match all the projects you need – In this case, since we need to retun this to vRA Custom Forms, mostly in drop-down form, we will be using string arrays
One configuration element per project, with the variables mentioned
  • An action that will return the values to the custom forms, using two inputs, the Project we want to get the information from, and the value that we want to get. That makes the action reusable by multiple fields in the form: In this case, I called it getConfigurationElementValue and it can be seen on the following link: https://github.com/luchodelorenzi/scripts/blob/master/getConfigurationElementValue.js
  • An action or workflow that will:
    • Get the data from the external API
    • Populate the configuration elements with that data

For this example, since I don’t have any external API in my lab, I will use static arrays to demonstrate the point in the code: The action is called updateConfigurationElements and can be seen in the following link https://github.com/luchodelorenzi/scripts/blob/master/updateConfigurationElements.js

This action/workflow can be scheduled to run every minute, every 5 minutes, depending on the need.

The data will be persisted in the vRO DB so that’s why I’m calling this a ‘database’ instead of a cache, however, it could very well be called a ‘persistent cache’ since all it is doing is to make the data available to the user as fast as possible but not doing any processing.

This workflow runs every 5 minutes and updates the values on all the existing projects (Configuration Elements)

The important thing to note here is that there isn’t any processing from the Custom Form to the vRO configuration elements when the user requests a catalog item!. Getting the data directly from the vRO DB without any processing at request time is what is going to give us the fastest loading times. All the processing is done in the background, without none of the requesters noticing!

  • The last step is to refer to the getConfigurationElementValue action in our custom form
    • A small caveat – the way vRA 8 and the ‘Project’ field works is that even though the project shows the user the names to be chosen, it is actually processing the IDs, so in this case I added a hidden field called ProjectName which is what I will be actually using to convert the IDs to names (since the configuration elements are based on the name)
Mapping the Project IDs to names
Using the getConfigurationElementValue action to get the values needed in the form

This is a small demo of how this works, take a look at the loading times for the form and changing the project! (And this is on a nested lab!)

Video Demo

Summary

To reiterate, the important things are:

  • No (or as little processing as possible if there is a field that cannot be used with configuration elements) should be done in the actions that are returning the data to the custom form
  • All the data should be processed in the background – The requester won’t be aware of it
  • Adding new projects it is as simple as duplicating one of the existing ones and changing the name. The way the workflows and actions are coded in this example will always look for every project (configuration element) below the ‘Projects’ folder
  • Getting the data out of the vRO DB directly via configuration elements instead of going to external sources, is the fastest way to get the values in the form.

Closing Note

I hope you found this interesting! It is using configuration elements in a way that might not be the most common usage, but it can bring great benefits to user experience when interacting with vRA requests. Having the data processed in the background and having really short form loading times will give the sensation of having more ‘speed’ to the tool itself!

Feel free to share this or leave a comment if you thought it was interesting!

Until next time!

Using ABX to change DNS Servers for a vRA Deployment at provisioning time

Hello Everyone,

On today’s post, we will go through creating an ABX Action to change the DNS Servers for a Deployment in vRA8. This might be needed to do in scenarios in which, even though the network has DNS servers configured, a specific deployment might require to use other DNS Servers while still being on the same network, for example, to join a different AD domain

The same idea can be used to edit other fields of the deployment, such as the IP Address, search domains, etc.

The post will be divided in 5 sections:

  • Cloud Template
  • Event Topic
  • ABX Action
  • ABX Subscription
  • Test

Cloud Template

In the template we’re going to need two inputs – dnsServers (comma separated list of DNS Servers) as well as an input to manage the amount of VMs in the deployment, we can call it ‘instances’

 instances:
    type: integer
    title: Amount of VMs
    default: 1
    maximum: 5
  dnsServers:
    type: string
    description: Comma separated list of DNS Servers
    title: DNS Servers
    default: '1.1.1.1,8.8.8.8'

These two values will be custom properties on the VM Object

properties:
   count: '${input.instances}'
   dnsServers: '${input.dnsServers}'

In addition to this, the network assignment for the VM resource should be set to ‘static’. A customization specification profile is optional, since using a ‘static’ assignment will auto-generate a ‘ghost’ customization specification profile at the time of provisioning

networks:
    - network: '${resource.VMNetwork1.id}'
      assignment: static

Event Topic

The event topic that we need to use to make changes to the Network Configuration is the one that has the object that we need to edit being presented to the workflow in an editable state, as in, not read-only.

For this specific use, the state is Network Configure

Pay special attention to the ‘Fired once for a cluster of machines’ part

The dnsServers object is a 3D Array, so that is what we need to use in the ABX Action Code

So from this point we learn that:

  • The action will run once for a cluster of machines, so if we were to do a Multi-VM deployment we need to take this into account, otherwise, it will only run for a single VM and not all of the VMs in the deployment
  • a 3D array needs to be used to insert the DNS Servers into the object at the event topic

ABX Action

For this example, I will use Python, and I will not use any external libraries for array management (such as numpy) since I wanted to see if it could be done natively. Python has way better native support for lists than it does for arrays, but in this case, given the schema of the object in the event topic, we’re forced to use a 3D Array.

The first thing we need to do when creating the action, is to add the needed inputs. In this one, I will add the custom properties of the resources as an input

Adding the custom properties

Once we have that as an input, we can use it to get the data we need (amount of instances and DNS servers)

To pass data back to the provisioning state, we will use and return the ‘outputs’ object

This is the code of the action itself, I will explain it below

def handler(context, inputs):
    outputs = {}
    dnsServers = [[[]]]
    instances = inputs["customProperties"]["count"]
    inputDnsServers = inputs["customProperties"]["dnsServers"].split(",")
    if (len(inputDnsServers) > 0):
        outputs["dnsServers"] = dnsServers
        outputs["dnsServers"][0][0] = inputDnsServers
        for i in range(1,int(instances)):
            outputs["dnsServers"] = outputs["dnsServers"] + [[inputDnsServers]]
    return outputs
  • Define the outputs object
  • Define the 3D Array for DNS Servers
  • Assign the inputs as variables in the script
  • Convert the comma separated string into a List
  • If the list is not empty (this means that the user did enter a value in the DNS Servers list on the input form) then we add the 3D Array to the outputs object.
    • Why am I asking to see if it is empty? Because if the user did not put anything on the field, we will be overwriting the output with an empty array, and that will overwrite the DNS that were read from the network in vRA. We only want to overwrite that if we’re actually changing the DNS Servers.
  • Also in the same condition, we want to add the DNS Servers array to each VM, so we iterate through the amount of VMs.
    • The way to add it without using numpy (we have no append method) is not elegant, but it does the trick. Basically, we initialize the first element and then we add other elements to the same array using the same format.
  • Return the outputs object

This can also be done in javascript and powershell, the idea would be the same.

So how does this object look like in an actual run?

In this example, I changed the DNS for 3 VMs – You can see that we’re using the 3D Array Structure

Lastly, we need to configure a subscription for it to run at this specific state.

ABX Subscription

This is the most straightforward part – We create a blocking subscription in the Network Configure state, and we add the action we just created

The ABX subscription can be filtered via a condition (for example, to run only on specific cloud templates) as well.

So let’s do our deployment now!

The network i’m going to select has the 8.8.8.8 DNS configured

This will be overwritten by whatever we put on the input form. I’m going to use 1.2.3.4 and 5.6.7.8 for this example, and there will be 2 VMs in the deployment

We can check the output of the action before the deployment finishes

Action run started by an Event (Subscription)
Action output

In there we can see the actual code that run, if it was successful or not, the payload, and the output the action had. In this case, our two DNS Servers for our two VMs with a successful output.

Checking the DNS for one of the VMs, we can see the two DNS Servers we selected as inputs!

Success!!!

Summary

I hope you found this post useful! The same idea can be used to change several other properties at provisioning time. Also, it was a learning experience for me to learn how to play with arrays in Python natively, and how to interact with ABX properly.

More posts will be coming soon!

If you liked this one, please share it and/or leave a comment!

Configuring a Dynamic Multi-NIC Cloud Template in vRA 8.x

Hello Everyone,

On today’s post, I will focus on a Dynamic Multi-NIC configuration for a Cloud Template in vRA 8.x

This allows customers to reuse the same cloud templates for virtual machines that could have a different amount of NICs, and this amount is defined at the time of the request. If this wasn’t dynamic, then a cloud template with three networks, will always need to have three networks configured at the time of the request, which might not be the case.

Using a Dynamic construct allows for less cloud template sprawl, since multiple application configurations can use the same cloud template.

Since this configuration is not trivial, this post will be a step by step guide on how to achieve this result.

Current Environment

For this Lab demonstration, we will use a vSphere Cloud Account, 4 NSX-T segments that are part of a Network Profile with a capability tag named “env:production” – In doing so, when using that constraint tag in the cloud template, we can guarantee our deployment will use that specific network profile.

The 4 NSX-T segments also have a single tag that refers to the type of network it is. In this scenario, Application, Frontend, Database and Backup are our 4 networks.

NSX-T Segments tagged and defined in the network profile
‘env:production’ tag in the network profile

Creating the Cloud Template

To get the Dynamic Multi-NIC configuration on the Cloud Template to work, we need the following things:

  • Inputs for Network to NIC mapping based on tagging
  • Inputs for NIC existence
  • Network Resources
  • VM Resource and Network Resource assignment

In addition to this, we can do customization in Service Broker to change the visibility of the fields. This is done to only allow the requester to choose a network mapping for a NIC what will actually be used.

Inputs for Network to NIC mapping based on tagging

This cloud template will allow for configurations of up to 4 NICs, and since we have 4 networks, we should let the requester select, for each NIC, what networks can be used.

This is what it looks like

Network1:
    type: string
    description: Select Network to Attach to
    default: 'net:application'
    title: Network 1
    oneOf:
      - title: Application Network
        const: 'net:application'
      - title: Frontend Network
        const: 'net:frontend'
      - title: Database Network
        const: 'net:database'
      - title: Backup Network
        const: 'net:backup'
  Network2:
    type: string
    description: Select Network to Attach to
    default: 'net:frontend'
    title: Network 2
    oneOf:
      - title: Application Network
        const: 'net:application'
      - title: Frontend Network
        const: 'net:frontend'
      - title: Database Network
        const: 'net:database'
      - title: Backup Network
        const: 'net:backup'
  Network3:
    type: string
    description: Select Network to Attach to
    default: 'net:database'
    title: Network 3
    oneOf:
      - title: Application Network
        const: 'net:application'
      - title: Frontend Network
        const: 'net:frontend'
      - title: Database Network
        const: 'net:database'
      - title: Backup Network
        const: 'net:backup'
  Network4:
    type: string
    description: Select Network to Attach to
    default: 'net:backup'
    title: Network 4
    oneOf:
      - title: Application Network
        const: 'net:application'
      - title: Frontend Network
        const: 'net:frontend'
      - title: Database Network
        const: 'net:database'
      - title: Backup Network
        const: 'net:backup'

We can see that each of the inputs allows for any of the networks to be selected.

Inputs for NIC Existence

Other than the first NIC (which should always exist, otherwise our VM(s) wouldn’t have any network connectivity, we want to be able to deploy VMs with 1, 2, 3, and 4 NICs, using the same Cloud Template.

To achieve that, we will create 3 Boolean Inputs that will define if a NIC should be added or not.

needNIC2:
    type: boolean
    title: Add 2nd NIC?
    default: false
  needNIC3:
    type: boolean
    title: Add 3rd NIC?
    default: false
  needNIC4:
    type: boolean
    title: Add 4th NIC?
    default: false

Network Resources

To manage the configuration of the NICs and networks, the network resources for NICs 2, 3 and 4 will use a count property, and this property’s result (either 0 if it doesn’t exist, or 1 if it does exist) will be based on the result of the inputs. Network 1 will not use that property

Also, we will use the deviceIndex property to maintain consistency with the numbering – So the network resources look like this

Network1:
    type: Cloud.vSphere.Network
    properties:
      networkType: existing
      deviceIndex: 0
      constraints:
        - tag: '${input.Network1}'
        - tag: 'env:production'
  Network2:
    type: Cloud.vSphere.Network
    properties:
      networkType: existing
      count: '${input.needNIC2 == true ? 1 : 0}'
      deviceIndex: 1
      constraints:
        - tag: '${input.Network2}'
        - tag: 'env:production'
  Network3:
    type: Cloud.vSphere.Network
    properties:
      networkType: existing
      count: '${input.needNIC3 == true ? 1 : 0}'
      deviceIndex: 2
      constraints:
        - tag: '${input.Network3}'
        - tag: 'env:production'
  Network4:
    type: Cloud.vSphere.Network
    properties:
      networkType: existing
      count: '${input.needNIC4 == true ? 1 : 0}'
      deviceIndex: 3
      constraints:
        - tag: '${input.Network4}'
        - tag: 'env:production'

The constraint tags that are used are the Network Input (to choose a network) and the ‘env:production’ tag to make our deployment use the Network Profile we defined earlier.

VM Resource & Network Resource Assignment

This is the tricky part – Since our networks could be non-existent (if the needNic input is not selected) we cannot use the regular syntax to add a network, which would be something like:

networks:
        - network: '${resource.Network1.id}'
          assignment: static
          deviceIndex: 0
        - network: '${resource.Network2.id}'
          assignment: static
          deviceIndex: 1
      ...

This will fail on the Cloud Template validation because the count for Network2 could be zero, so to do the resource assignment, we need to use the map_by syntax.

Several other examples can be seen on the following link: https://docs.vmware.com/en/vRealize-Automation/8.5/Using-and-Managing-Cloud-Assembly/GUID-12F0BC64-6391-4E5F-AA48-C5959024F3EB.html

The VM resource uses a simple Ubuntu Image with a Small Flavor, so here is what it looks like once the map_by syntax is used for the assignment

Cloud_vSphere_Machine_1:
    type: Cloud.vSphere.Machine
    properties:
      image: Ubuntu-TMPL
      flavor: Small
      customizationSpec: Linux
      networks: '${map_by(resource.Network1[*] + resource.Network2[*] + resource.Network3[*] + resource.Network4[*], r => {"network":r.id, "assignment":"static", "deviceIndex":r.deviceIndex})}'
      constraints:
        - tag: 'env:production'

28/07/22 Update

I’ve gotten comments saying this didn’t work in newer versions such as vRA 8.7 or 8.8. The syntax for those versions might be:

networks: '${map_by(resource.Network1[*].r + resource.Network2[*].r + resource.Network3[*].r + resource.Network4[*].r, r => {"network":r.id, "assignment":"static", "deviceIndex":r.deviceIndex})}'

This allows for any combination of NICs, from 1 to 4, and if the count of one of the resources is 0, it won’t be picked up by the assignment expression.

This is what the Cloud Template looks on the canvas. You can see that Networks 2, 3 and 4 have the appearance of possible multiple instances. This is because we’re using the count parameter.

Canvas view of the Cloud Template

If we were to deploy this Cloud Template, it will look like this:

Doesn’t make much sense to select networks that we won’t assign, right?

How do we fix this? We can leverage Service Broker to manage the visibility of the fields based on the boolean input!

Using the inputs as conditional value for the visibility of the network field

So now, from Service Broker, it looks like this:

No extra NICs selected
NICs 2 and 3 selected

So if we deploy this, it should have three networks assigned. The first NIC should use the Application Network, the second NIC should use the Frontend Network and the 3rd NIC should use the Database Network.

Let’s test it!

TA-DA!

We can see that even if the Cloud Template had 4 Network Resources, only 3 were instantiated for this deployment! And each network was mapped to a specific NSX-T segment, thanks to the constraint tags.

Closing Note

I hope this blog post was useful – The same assignment method can be used for other resources such as Disks or Volumes – the principle is still the same.

Feel free to share this if you found it useful, and leave your feedback in the comments.

Until the next time!

Updating an Onboarded Deployment in vRA 8.x

Hello Everyone!

On today’s post, we will go through the process of updating an onboarded deployment in vRA 8.x

The onboarding feature allows customers to add VMs that were not deployed from vRA, to the vRA infrastructure. This means that these VMs are added to one or more deployments, and once they exist within vRA, operations such as power cycling, opening a remote console, or resizing CPU/RAM are now available.

However, there are scenarios in which customers would want to expand these deployments, not with new onboarded VMs, but with newly deployed VMs (or other resources) from vRA! These deployments will use an image, a flavor, could use a multitude of inputs, tagging, networks, etc. So how do we do this?

Onboarding the VMs using an auto-generated Cloud Assembly Template

The first thing we need to do, is to create an onboarding plan, select a name for our deployment, and select the VMs we’re going to onboard initially.

Creating the Onboarding Plan
Adding two VMs to be onboarded

On the deployments tab, we can rename the deployment if needed, but the most important part is to select Cloud Template Configuration and change it to Create Cloud Template in Cloud Assembly Format this will allow us to have a source for our deployment, that we can edit afterwards to allow for future growth

Cloud Template in Cloud Assembly format

It is important to note that the imageRef has no image available. Since this is not a vRA Deployment but an Onboarding, none of the resources are being deployed from any of the images. We will come back to this item later.

After saving this configuration and clicking on Run, our deployment will be onboarded

Updating the onboarded deployment to add a new VM in a specific network

If we check on the onboarded deployment, we will see that it is mapped to a specific Cloud Template (the one that was auto-generated earlier by the Onboarding Plan)

So if we were to do an update on this deployment, we need to edit that Cloud Template

I will now add a vSphere Machine resource as well as a vSphere Network:

inputs: {}
resources:
  Cloud_vSphere_Machine_1:
    type: Cloud.vSphere.Machine
    properties:
      image: 'ubuntu'
      cpuCount: 1
      totalMemoryMB: 1024
      networks:
        - network: '${resource.Cloud_vSphere_Network_1.id}'
  Cloud_vSphere_Network_1:
    type: Cloud.vSphere.Network
    properties:
      networkType: existing
      constraints: 
        - tag: env:vsphere  
  DevTools-02a:
    type: Cloud.vSphere.Machine
    properties:
      imageRef: no_image_available
      cpuCount: 1
      totalMemoryMB: 4096
  DevTools-01a:
    type: Cloud.vSphere.Machine
    properties:
      imageRef: no_image_available
      cpuCount: 1
      totalMemoryMB: 4096
  

This is what our template looks like now. So the next thing we should do is click on Update, right?

Update is Greyed out!

The update task is greyed out because ir Cloud Template does not have inputs. Since we don’t have inputs, what we need to do is to go to the Cloud Template, and instead of selecting Create a New Deployment we should select Update an Existing Deployment and then click on the onboarded deployment.

Updating the Onboarded Deployment

After clicking on Next, the plan is presented.

Notice something wrong here?

The update operation will attempt to re-create the onboarded VMs! That’s not something we want, and also, in this scenario, it will fail since there is no image mapping to deploy from!

What we want is to leave all the VMs that were previously onboarded, untouched, and only add our new VM and network. So how do we achieve this?

This is achieved by adding the ignorechanges parameter with a value of true to every resource in the cloud template that was previously onboarded – In this scenario, this would be our 2 DevTools VMs

Adding the ignoreChanges parameter

If we re-try updating the deployment now, the only tasks that should appear will be the ones for the new resources (VM and Network)

Update deployment showing the new tasks

After clicking on ‘deploy’ and waiting for it to finish, our deployment will now like this

Deployment updated with our new VM and network! Hooray!

Offboarding/Unregistering limitations

It is important to note that vRA’s limitations for unregistering VMs are still present. The only VMs that can be unregistered from vRA are the ones that were previously onboarded. VMs that were deployed from vRA will not be able to be unregistered without deletion. The fact that the deployment VMs are part of an Onboarded Deployment does not change this.

Closing Note

I hope you enjoyed this post! When I started working on this use case I figured it was not as trivial as I thought, and after doing research and testing, found this walkthrough/solution.

Let me know if this was useful in the comments!

Until next time!

Deploying a non-standard VCF 4.2 Workload Domain via API!

Getting started with VMware Cloud Foundation (VCF) - CormacHogan.com

Hello Everyone!

On today’s post, as a continuation of the previous post (in which we talk about the VCF MGMT Domain) I will show a step by step guide of how to do a complete deployment of a VCF Workload Domain, subject to some specific constraints based on a project I was working on, using VCF’s API!

What’s this non-standard architecture like?

In this specific environment, I had to play around the following constraints

  • 4 hosts with 256GB of RAM using vSAN, check the previous post for information about the MGMT domain!
  • 3 Hosts with 256GB of RAM, using vSAN
  • 3 Hosts with 1.5TB of RAM, using FC SAN storage
  • Hosts using 4×10 NICs
  • NIC Numbering not being consistent (some hosts had 0,1,2,3 – other hosts had 4,5,6,7 – even though this can be changed editing files on the ESXi, it is still a constraint and can be worked around using the API)

With this information, the decision was to:

  • Separate the Workload Domain into 2 clusters, one for NSX-T Edges and the other one for Compute workloads, given the discrepancies in RAM and storage configuration, they could never be part of the same logical cluster.

This looks something like…

It is impossible to deploy this using the GUI, due to the following:

  • Can’t utilize 4 Physical NICs for a Workload Domain
  • Can’t change NIC numbering or NIC to DVS uplink mapping

So we have to do this deployment using the API! Let’s go!

Where do we start?

First of all, VCF’s API documentation is public, and this is the link to it: https://code.vmware.com/apis/1077/vmware-cloud-foundation – I will be referring to this documentation A LOT over the course of this blog post

All the API calls require the use of a token, which is generated with the following request (example taken from the documentation)

cURL Request

$ curl 'https://sfo-vcf01.rainpole.io/v1/tokens' -i -X POST \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d '{
  "username" : "administrator@vsphere.local",
  "password" : "VMware123!"
}'

Once we have the token, we can use it in other API calls until it expires and we just either refresh it or create a new one. All the VCF API calls that are generated to SDDC manager (not internal API calls) will require the usage of a bearer token.

List of steps to create a workload domain

  • Commission all hosts from SDDC manager and create network profiles appropriately to match the external storage selection – In this scenario, we will have a network profile for the vSAN based hosts, as well as another network profile for the FC SAN based hosts. Hosts can also be commissioned via API calls (3.65 in the API reference) instead of doing it via the GUI, but the constraints I had did not prevent me from doing it via GUI.
  • Get all the IDs for the commisioned hosts – The API Call is “2.7.2 Get the Hosts” and it is a GET call to https://sddc_manager_url/v1/hosts using Bearer Token authentication
  • Create the Workload Domain with a single cluster (Compute) – The API Call is “2.9.1 Create a Domain”
  • Add the Secondary Cluster (Edge) to the newly-created workload domain – The API Call is “2.10.1 Create a Cluster”
  • Create the NSX-T Edge Cluster on top of the Edge Cluster – The API Call is “2.37.3 – Create Edge Cluster”

For each of these tasks, we should first validate our JSON body before executing the API call. We will discuss this further.

You might ask, why don’t you create a Workload Domain with two clusters instead of first creating the Workload Domain with a single cluster and then adding the second one?

This is something I hit during the implementation – If we check the Clusters object on the API, we can see it is an array, so it should be able to work with multiple cluster values.

"computeSpec": { "clusterSpecs": [

The info on the API call also points to the fact that we should be able to create multiple clusters on the “Create Domain” call.

Even worse, the validation API will validate an API call with multiple clusters

However, I came to learn (after trying multiples times and contacting the VCF Engineering team, that this is not the case)

For example, if our body looked something like this (with two clusters), the validation API will work!

"computeSpec": {
      "clusterSpecs": [
        {
          "name": "vsphere-w01-cl-01",
          "hostSpecs": [
            {
              "id": "b818ba18-2960-49ce-a876-ed4e0c07a936",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic0",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic1",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic2",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  },
                  {
                    "id": "vmnic3",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  }
                ]
              }
            },
            {
              "id": "bd152a18-7b31-4cd4-a352-b94a7119bb33",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic0",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic1",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic2",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  },
                  {
                    "id": "vmnic3",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  }
                ]
              }
            },
            {
              "id": "18409da3-fbae-47b2-800f-67d032fe21a0",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic0",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic1",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic2",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  },
                  {
                    "id": "vmnic3",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  }
                ]
              }
            }
          ],
          "datastoreSpec": {
            "vmfsDatastoreSpec" : {
              "fcSpec" : [ {
              "datastoreName" : "vsphere-m01-fc-datastore1"
             } ]
             }
          },
          "networkSpec": {
            "vdsSpecs": [
              {
                "name": "vsphere-w01-cl01-vds01",
                "portGroupSpecs": [
                  {
                    "name": "vsphere-w01-cl01-vds-pg-mgmt",
                    "transportType": "MANAGEMENT"
                  },
                  {
                    "name": "vsphere-w01-cl01-vds-pg-vmotion",
                    "transportType": "VMOTION"
                  }
                ]
              },
              {
                "name": "vsphere-w01-cl01-vds02",
                "isUsedByNsxt": true
              }
            ],
            "nsxClusterSpec" : {
            "nsxTClusterSpec" : {
              "geneveVlanId" : 1214,
              "ipAddressPoolSpec" : {
                "name" : "vsphere-w01-np01",
                "subnets" : [ {
                "ipAddressPoolRanges" : [ {
                  "start" : "172.22.14.100",
                  "end" : "172.22.14.200"
                } 
              ],
                "cidr" : "172.22.14.0/24",
                "gateway" : "172.22.14.254"
                } ]
               }
             }
            }
          }
        },
          {
          "name": "vsphere-w01-cl-edge-01",
          "hostSpecs": [
            {
              "id": "aa699b0d-015f-43e9-83ea-6e941b37e642",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic4",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic5",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic6",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  },
                  {
                    "id": "vmnic7",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  }
                ]
              }
            },
            {
              "id": "1e500b1b-fd33-425c-8c6d-42840cf658db",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic4",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic5",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic6",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  },
                  {
                    "id": "vmnic7",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  }
                ]
              }
            },
            {
              "id": "e138d6a1-6c55-4326-ac6c-ffc0239e15b5",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic4",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic5",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic6",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  },
                  {
                    "id": "vmnic7",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  }
                ]
              }
            }
          ],
          "datastoreSpec": {
            "vsanDatastoreSpec": {
              "failuresToTolerate": 1,
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "datastoreName": "vsphere-w01-ds-vsan-01"
            }
          },
          "networkSpec": {
            "vdsSpecs": [
              {
                "name": "vsphere-w01-cl-edge-01-vds01",
                "portGroupSpecs": [
                  {
                    "name": "vsphere-w01-cl-edge-01-pg-mgmt",
                    "transportType": "MANAGEMENT"
                  },
                  {
                    "name": "vsphere-w01-cl-edge-01-pg-vsan",
                    "transportType": "VSAN"
                  },
                  {
                    "name": "vsphere-w01-cl-edge-01-pg-vmotion",
                    "transportType": "VMOTION"
                  }
                ]
              },
              {
                "name": "vsphere-w01-cl-edge-01-vds02",
                "isUsedByNsxt": true
              }
            ],
            "nsxClusterSpec" : {
                "nsxTClusterSpec" : {
                  "geneveVlanId" : 1214,
                  "ipAddressPoolSpec" : {
                      "name" : "vsphere-w01-np02",
                      "subnets" : [ {
                        "ipAddressPoolRanges" : [ {
                          "start" : "172.22.14.210",
                          "end" : "172.22.14.230"
                        } 
                      ],
                        "cidr" : "172.22.14.0/24",
                        "gateway" : "172.22.14.254"
                        } ]
                    }
                      
                }
            }
           }
        }
      ]
    },

However, when we go ahead and try to create it, it will fail, and we will see the following error on the logs.

ERROR [vcf_dm,02a04e83325703b0,7dc4] [c.v.v.v.c.v1.DomainController,http-nio-127.0.0.1-7200-exec-6]  Failed to create domain
com.vmware.evo.sddc.common.services.error.SddcManagerServicesIsException: Found multiple clusters for add vi domain.
at com.vmware.evo.sddc.common.services.adapters.workflow.options.WorkflowOptionsAdapterImpl.getWorkflowOptionsForAddDomainWithNsxt(WorkflowOptionsAdapterImpl.java:1222)

So, as mentioned earlier, we need to first create our domain (with a single cluster), and then add the 2nd cluster!

1: Create a Workload Domain with a Single Cluster

We will first create our Workload Domain with the compute cluster, which in this scenario, uses external storage, and will use the secondary distributed switch for overlay traffic.

This is my API call body based on the API reference, to create a Workload Domain with a single cluster of 3 hosts, using two VDS, 4 physical NICs numbered from 0 to 3 and external FC storage, using the host IDs that I got after the previous step.

{
    "domainName": "vsphere-w01",
    "orgName": "vsphere.local",
    "vcenterSpec": {
      "name": "vsphere-w01-vc01",
      "networkDetailsSpec": {
        "ipAddress": "172.22.11.64",
        "dnsName": "vsphere-w01-vc01.vsphere.local",
        "gateway": "172.22.11.254",
        "subnetMask": "255.255.255.0"
      },
      "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
      "rootPassword": "VMware1!",
      "datacenterName": "vsphere-w01-dc-01"
    },
    "computeSpec": {
      "clusterSpecs": [
        {
          "name": "vsphere-w01-cl-01",
          "hostSpecs": [
            {
              "id": "b818ba18-2960-49ce-a876-ed4e0c07a936",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic0",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic1",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic2",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  },
                  {
                    "id": "vmnic3",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  }
                ]
              }
            },
            {
              "id": "bd152a18-7b31-4cd4-a352-b94a7119bb33",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic0",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic1",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic2",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  },
                  {
                    "id": "vmnic3",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  }
                ]
              }
            },
            {
              "id": "18409da3-fbae-47b2-800f-67d032fe21a0",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic0",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic1",
                    "vdsName": "vsphere-w01-cl01-vds01"
                  },
                  {
                    "id": "vmnic2",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  },
                  {
                    "id": "vmnic3",
                    "vdsName": "vsphere-w01-cl01-vds02"
                  }
                ]
              }
            }
          ],
          "datastoreSpec": {
            "vmfsDatastoreSpec" : {
              "fcSpec" : [ {
              "datastoreName" : "vsphere-m01-fc-datastore1"
             } ]
             }
          },
          "networkSpec": {
            "vdsSpecs": [
              {
                "name": "vsphere-w01-cl01-vds01",
                "portGroupSpecs": [
                  {
                    "name": "vsphere-w01-cl01-vds-pg-mgmt",
                    "transportType": "MANAGEMENT"
                  },
                  {
                    "name": "vsphere-w01-cl01-vds-pg-vmotion",
                    "transportType": "VMOTION"
                  }
                ]
              },
              {
                "name": "vsphere-w01-cl01-vds02",
                "isUsedByNsxt": true
              }
            ],
            "nsxClusterSpec" : {
            "nsxTClusterSpec" : {
              "geneveVlanId" : 1214,
              "ipAddressPoolSpec" : {
                "name" : "vsphere-w01-np01",
                "subnets" : [ {
                "ipAddressPoolRanges" : [ {
                  "start" : "172.22.14.100",
                  "end" : "172.22.14.200"
                } 
              ],
                "cidr" : "172.22.14.0/24",
                "gateway" : "172.22.14.254"
                } ]
               }
             }
            }
          }
        }
      ]
    },
    "nsxTSpec": {
      "nsxManagerSpecs": [
        {
          "name": "vsphere-w01-nsx01a",
          "networkDetailsSpec": {
            "ipAddress": "172.22.11.76",
            "dnsName": "vsphere-w01-nsx01a.vsphere.local",
            "gateway": "172.22.11.254",
            "subnetMask": "255.255.255.0"
          }
        },
        {
          "name": "vsphere-w01-nsx01b",
          "networkDetailsSpec": {
            "ipAddress": "172.22.11.77",
            "dnsName": "vsphere-w01-nsx01b.vsphere.local",
            "gateway": "172.22.11.254",
            "subnetMask": "255.255.255.0"}
        },
        {
          "name": "vsphere-w01-nsx01c",
          "networkDetailsSpec": {
            "ipAddress": "172.22.11.78",
            "dnsName": "vsphere-w01-nsx01c.vsphere.local",
            "gateway": "172.22.11.254",
            "subnetMask": "255.255.255.0"}
        }
      ],
      "vip": "172.22.11.75",
      "vipFqdn": "vsphere-w01-nsx01.vsphere.local",
      "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
      "nsxManagerAdminPassword": "VMware1!VMware1!"
    }
  }

Important!

  • The DVS that is going to be used for overlay traffic must have the isUsedByNsxt flag set to true. In the case of a 4 NIC and 2 VDS deployment such as this one, it shouldn’t have any of the management, vMotion or vSAN traffic.

With the body, to execute the VALIDATE and EXECUTE api calls, we will do the following: (high level overview since we can use any REST API tool such as Postman, curl, invoke-restmethod, or any wrapper from any language that can execute REST calls)

The list of steps will be the same for all the POST API calls, changing the URL to match each specific call.

If the validation is successful, we will get a message similar to:

 "description": "Validating Domain Creation Spec",
    "executionStatus": "COMPLETED",
    "resultStatus": "SUCCEEDED",
    "validationChecks": [
        {
            "description": "DomainCreationSpecValidation",
            "resultStatus": "SUCCEEDED"
        }

We should continue editing and retrying in case of errors until we get the validation to pass, do not attempt to execute the API call without validating it first!

Once the validation has passed, we can follow the same steps that are mentioned above but instead of making a POST call to https://sddc_manager_fqdn/v1/domains/validations, we remove the “validations” part, so it would be a call to https://sddc_manager_fqdn/v1/domains.

The deployment will start and after a couple minutes we will see in the SDDC console that it was successful.

If it were to fail for whatever reason, we can troubleshoot the deployment by checking where it failed on the SDDC console as well as checking logs, but as long as the validation passes, it should not be a problem with the body we’re sending.

2: Adding a 2nd Cluster to the existing workload domain

To add a cluster to an existing domain, the first thing we need is to get the ID of the domain, that can easily be done with a GET call to https://sddc_manager_url/v1/domains and selecting the ID of the workload domain we just created.

Once we get the ID, this is the body (following the API reference) to add a new cluster to an existing domain.

{
    "domainId": "58a6cdcb-f609-49dd-9729-7e27d65440c6",
    "computeSpec": {
      "clusterSpecs": [
          {
          "name": "vsphere-w01-cl-edge-01",
          "hostSpecs": [
            {
              "id": "aa699b0d-015f-43e9-83ea-6e941b37e642",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic4",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic5",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic6",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  },
                  {
                    "id": "vmnic7",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  }
                ]
              }
            },
            {
              "id": "1e500b1b-fd33-425c-8c6d-42840cf658db",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic4",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic5",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic6",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  },
                  {
                    "id": "vmnic7",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  }
                ]
              }
            },
            {
              "id": "e138d6a1-6c55-4326-ac6c-ffc0239e15b5",
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "hostNetworkSpec": {
                "vmNics": [
                  {
                    "id": "vmnic4",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic5",
                    "vdsName": "vsphere-w01-cl-edge-01-vds01"
                  },
                  {
                    "id": "vmnic6",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  },
                  {
                    "id": "vmnic7",
                    "vdsName": "vsphere-w01-cl-edge-01-vds02"
                  }
                ]
              }
            }
          ],
          "datastoreSpec": {
            "vsanDatastoreSpec": {
              "failuresToTolerate": 1,
              "licenseKey": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
              "datastoreName": "vsphere-w01-ds-vsan-01"
            }
          },
          "networkSpec": {
            "vdsSpecs": [
              {
                "name": "vsphere-w01-cl-edge-01-vds01",
                "portGroupSpecs": [
                  {
                    "name": "vsphere-w01-cl-edge-01-pg-mgmt",
                    "transportType": "MANAGEMENT"
                  },
                  {
                    "name": "vsphere-w01-cl-edge-01-pg-vsan",
                    "transportType": "VSAN"
                  },
                  {
                    "name": "vsphere-w01-cl-edge-01-pg-vmotion",
                    "transportType": "VMOTION"
                  }
                ]
              },
              {
                "name": "vsphere-w01-cl-edge-01-vds02",
                "isUsedByNsxt": true
              }
            ],
            "nsxClusterSpec" : {
                "nsxTClusterSpec" : {
                  "geneveVlanId" : 1214,
                  "ipAddressPoolSpec" : {
                      "name" : "vsphere-w01-np02",
                      "subnets" : [ {
                        "ipAddressPoolRanges" : [ {
                          "start" : "172.22.14.210",
                          "end" : "172.22.14.240"
                        } 
                      ],
                        "cidr" : "172.22.14.0/24",
                        "gateway" : "172.22.14.254"
                        } ]
                    }
                      
                }
            }
           }
        }
      ]
    }
  }

Even though we don’t need the cluster to be prepared for NSX-T (since it will only be used for Edges) setting the isUsedByNSXT flag to true will make the secondary VDS used by the uplink portgroups once we create a T0, which is what we want in this scenario – otherwise, we would not be using the 3rd and 4th NICs at all.

As discussed earlier, we should first run the POST call to validate in this case, the URL is https://sddc_manager_fqdn/v1/clusters/validations and after the body is validated, proceed with the creation removing validation from the URL

Last but not least, we need to create our NSX-T Edge Cluster on top of the 2nd cluster on the domain!

3: Create NSX-T Edge Cluster

The last piece of the puzzle is creating the NSX-T Edge Cluster, to allow for this workload domain to leverage overlay networks and communicate to the physical world.

To create the NSX-T Edge Cluster, we first need to get the Cluster ID of the cluster we just created (how many times can you say cluster in the same sentence?)

Following the API reference, number 2.10.1 is ‘Get Clusters’, which does a GET call to https://sddc_manager_fqdn/v1/clusters

Now that we have the ID, this is the body to create two Edge Nodes, configure management, TEP and uplink interfaces, configure a T0 and a T1 instance, as well as configuring BGP peering on the T0 instance!

{
    "edgeClusterName" : "vsphere-w01-ec01",
    "edgeClusterType" : "NSX-T",
    "edgeRootPassword" : "VMware1!VMware1!",
    "edgeAdminPassword" : "VMware1!VMware1!",
    "edgeAuditPassword" : "VMware1!VMware1!",
    "edgeFormFactor" : "LARGE",
    "tier0ServicesHighAvailability" : "ACTIVE_ACTIVE",
    "mtu" : 9000,
    "asn" : 65212,
    "edgeNodeSpecs" : [ {
      "edgeNodeName" : "vsphere-w01-en01.vsphere.local",
      "managementIP" : "172.22.11.71/24",
      "managementGateway" : "172.22.11.254",
      "edgeTepGateway" : "172.22.17.254",
      "edgeTep1IP" : "172.22.17.12/24",
      "edgeTep2IP" : "172.22.17.13/24",
      "edgeTepVlan" : 1217,
      "clusterId" : "37c83ee6-2338-40b0-9470-bb6d47922601",
      "interRackCluster" : false,
      "uplinkNetwork" : [ {
        "uplinkVlan" : 1218,
        "uplinkInterfaceIP" : "172.22.18.2/24",
        "peerIP" : "172.22.18.1/24",
        "asnPeer" : 65213,
        "bgpPeerPassword" : "VMware1!"
      }, {
        "uplinkVlan" : 1219,
        "uplinkInterfaceIP" : "172.22.19.2/24",
        "peerIP" : "172.22.19.1/24",
        "asnPeer" : 65213,
        "bgpPeerPassword" : "VMware1!"
      } ]
    }, {
        "edgeNodeName" : "vsphere-w01-en02.vsphere.local",
        "managementIP" : "172.22.11.72/24",
        "managementGateway" : "172.22.11.254",
        "edgeTepGateway" : "172.22.17.254",
        "edgeTep1IP" : "172.22.17.14/24",
        "edgeTep2IP" : "172.22.17.15/24",
        "edgeTepVlan" : 1217,
        "clusterId" : "37c83ee6-2338-40b0-9470-bb6d47922601",
        "interRackCluster" : false,
        "uplinkNetwork" : [ {
          "uplinkVlan" : 1218,
          "uplinkInterfaceIP" : "172.22.18.3/24",
          "peerIP" : "172.22.18.1/24",
          "asnPeer" : 65213,
          "bgpPeerPassword" : "VMware1!"
        }, {
          "uplinkVlan" : 1219,
          "uplinkInterfaceIP" : "172.22.19.3/24",
          "peerIP" : "172.22.19.1/24",
          "asnPeer" : 65213,
          "bgpPeerPassword" : "VMware1!"
      } ]
    } ],
    "tier0RoutingType" : "EBGP",
    "tier0Name" : "vsphere-w01-ec01-t0-gw01",
    "tier1Name" : "vsphere-w01-ec01-t1-gw01",
    "edgeClusterProfileType" : "DEFAULT"
  }

As mentioned before, please run the VALIDATE call first, in this scenario, a POST call to https://sddc_manager_fqdn/v1/edge-clusters/validations – after validation is passed, proceed to execute the call without the validations on the URL.

After this procedure is finished, we will have our workload domain with two clusters as well as a T0 gateway completely configured and ready to go! Simple and quick, isn’t it?

Closing Note

Leveraging APIs for VCF can help us not only to work with architectures or designs that are not able to be implemented due to GUI restrictions, but also greatly speed up the time we take in doing so!

I hope you enjoyed this post, and if you have any concerns, or want to share your experience deploying VCF via API calls, feel free to do so!

See you in the next post!

Lessons learned while deploying VCF 4.2 Management Domain

Hello Everyone! It’s me again, trying to maintain a weekly post cadence!

Today I’m going to talk about some roadblocks I hit while doing a 4.2 VCF Deployment in a real, customer environment. Hopefully this will prevent these issues from happening to you or help you to solve them quickly if they do arise!

Getting started with VMware Cloud Foundation (VCF) 4.0 - CormacHogan.com

Password Policy for Cloud Builder

In VCF 4.2, several changes to password strength were made. It seems that using 8 character passwords are hit/miss (you could get a valid deployment and then immediately a non-valid deployment if you deploy another Cloud Builder with a password like “VMw@r3!!” – I haven’t been able to fully grasp the cause for this behaviour.

In addition, VMware is now a dictionary word, so it wont be allowed. So “VMware1!” and “VMware1!VMware1!” will also fail.

The password that i’ve been using successfully for the initial deployment is “VMw@r3!!VMw@r3!!” – That one works 100% – You can go ahead and use that one.

Hostnames in uppercase

This one is really, really strange – If the hostnames of your ESXi hosts are in uppercase, you will get a ‘Failed to connect to lowercase_hostname’ for all of your hosts when running the validation, and the validation will stop and won’t query any of the host configuration

I spent some time trying to figure this out, at first I thought it was DNS records, but then on a different environment, 3 of the 4 hosts had their hostname in upper case and one of them in lower case, and the one in lower case was the only one connecting, so that made me test the change and suddenly the new host in lowercase was also connecting!

To clarify, ESXI1.VSPHERE.LOCAL will fail, esxi1.vsphere.local will work – Make sure your hostnames are in lowercase

Heterogeneous / Unbalanced disk configuration across hosts

This one is really interesting, let’s say you’re doing an all flash VCF and you have 20 disks per host – The best way to configure it would be 4 Disk groups of 1 Cache + 4 Capacity, so that you would use all 20 disks.

Since you can have at a maximum 5 Disk groups of 1 Cache + 7 Capacity, 40 is the maximum number of disks you can have.

However, make sure that you’re following these two rules for your deployment

  • Make sure that the amount of disks follows a multiple of a homogeneous disk group configuration so that all your disks can be used and all the disk groups have the same amount of disks – I.e, if you have 22 disks, there is no way you can use all disks while maintaining all disk groups with the same amount of disks. If you have 22 disks, you can do 3 (1+6) and one won’t be used, or 4(1+4) and two won’t be used.
  • Make sure that all your hosts have the same amount of disks. You can check this before installing – In my scenario, validation was passing but it was setting the cluster as hybrid instead of all flash.
    After checking that all devices were SSD and were marked as SSD I was really confused. Then I checked and two of the hosts had 2 more disks than the rest. Fixing that made the validation pass and marking the cluster as all flash.

EVC Mode

This one almost made me reinstall the whole cluster…

BE REALLY SURE that you’re selecting the correct EVC mode for your CPU family if you’re selecting an EVC mode in the Cloud Builder spreadsheet.

If you select the wrong EVC mode, Cloud Builder will fail in this deployment, and you won’t be able to continue from the GUI at all. The only way around it is via the API. Otherwise, it is wiping the cluster and starting from scratch!

I’m going to show you how to fix this issue but the method applies in case you need to edit the configuration and then re-attempt a deployment.

First of all, you need to get your SDDC Deployment ID, you can get it with this API call (I will be using curl for this example but you can also use something like invoke-restmethod in powershell or even a GUI based REST client such as Postman)

Get your SDDC Deployment ID

curl 'https://cloud_builder_fqdn/v1/sddcs/' -i -u 'admin:your_password' -X GET \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -k

You can export the output to a file or to a text viewing tool such as less, and then search for the sddcId value

Editing the JSON File

Once you have the sddcId, you need to edit the JSON file that CB generated from the spreadsheet so you can then use it in the API call. I recommend that you copy the file and edit the copy. The file is located at /opt/vmware/sddc-support/cloud_admin_tools/resources/vcf-public-ems/

#COPY THE FILE
cp /opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-public-e                                                                                                                     ms/vcf-public-ems.json /tmp/newjson.json
#REPLACE STRING ON FILE
sed -i "s/cascadelake/haswell/g" /tmp/newjson.json

You can also edit the file using vi – in this case I used sed because I knew the string will only appear once in the file and it was faster

Restarting the deployment

Now that you have the sddcId and you’ve edited the JSON file, it is time for you to restart the process using another API call

curl 'https://cloud_builder_fqdn/v1/sddcs/your_sddc_id_from_previous_step' -i -u 'admin:your_password' -X PATCH     -H 'Content-Type: application/json'     -H 'Accept: application/json'     -d "@/tmp/newjson.json"  -k

Make sure to add the @ before the location of the file when using curl

Once you run this, you should get something like:

HTTP/1.1 100 Continue
HTTP/1.1 200
Server: nginx
Date: Wed, 07 Apr 2021 20:37:08 GMT

And if you log in to the Cloud Builder web interface, your deployment should be running again! Phew, you saved yourself from reinstalling and preparing 4 nodes! Go grab a beer while the deployment continues 😀

Driver Issue when installing NSX-T VIBs

I ran into this issue after waiting for multiple hours for the NSX-T Host Preparation to finish, and seeing all the hosts on the NSX-T tab being marked as failed.

When checking the debug logs for Cloud Builder, I saw errors like:

2021-04-07T23:06:44.700+0000 [bringup,196c7022580bfc32,5a84] DEBUG [c.v.v.c.f.p.n.p.a.ConfigureNsxtTransportNodeAction,bringup-exec-7] TransportNode esxi1.vsphere.local DeploymentState state is {"details":[{"failureCode":260
80,"failureMessage":"Failed to install software on host. Failed to install software on host. esxi1.vsphere.local : java.rmi.RemoteException:  [DependencyError] VIB QLC_bootbank_qedi_2.19.9.0-1OEM.700.1.0.15843807 requires qe
dentv_ver \u003d X.40.17.0, but the requirement cannot be satisfied within the ImageProfile. VIB QLC_bootbank_qedf_2.2.8.0-1OEM.700.1.0.15843807 requires qedentv_ver \u003d X.40.17.0, but the requirement cannot be satisfied within the Im
ageProfile. Please refer to the log file for more details.","state":"failed","subSystemId":"eeaefa1e-c5a2-4a8a-9623-994b94a803a9","__dynamicStructureFields":{"fields":{},"name":"struct"}}],"state":"failed","__dynamicStructureFields":{"fi
elds":{},"name":"struct"}}

This is related to QLogic drivers that are included in the HP custom image that was being used in this deployment (and was patched to 7.0u1d which is the pre-requisite for VCF 4.2)

Indeed, these drivers were installed

esxcli software vib list | grep qed
qedf                           2.2.8.0-1OEM.700.1.0.15843807         QLC     VMwareCertified   2021-03-03
qedi                           2.19.9.0-1OEM.700.1.0.15843807        QLC     VMwareCertified   2021-03-03
qedentv                        3.40.3.0-12vmw.701.0.0.16850804       VMW     VMwareCertified   2021-03-04
qedrntv                        3.40.4.0-12vmw.701.0.0.16850804       VMW     VMwareCertified   2021-03-04

None of these drivers were in use, and none of the hosts were using QLogic hardware – So these drivers could be removed without issues, however, it is best to unconfigure the hosts from NSX-T first since that also prompts for a reboot.

Go to the Transport Node tab in NSX-T, select the cluster, and click on “Unprepare” – This will likely fail and prompt you to run a force cleanup – This one will work and the hosts will disappear from the tab.

In my scenario, none of the NSX-T VIBs were installed so no NSX-T VIB cleanup was necessary

Now, it is time to delete the drivers from the hosts and reboot them. You can run this one by one on the hosts (since you already have vCenter, vCLS, and NSX Manager VMs running, you can’t just blindly power-off all your hosts)

esxcli software vib remove --vibname=qedentv --force
esxcli software vib remove --vibname=qedrntv --force
esxcli software vib remove --vibname=qedf --force
esxcli software vib remove --vibname=qedi --force
esxcli system maintenanceMode set --enable true
esxcli system shutdown reboot --reason "Drivers"

Edge TEP to ESXi TEP validation when using Static IP Pool

VCF 4.2 removes the need of having a DHCP server on the ESXi TEP network (as long as you’re not using stretched cluster) which is a lifesaver for many, since setting up the DHCP server was usually a light stopper for customers (the other one being BGP)

However, the validation still attempts to search for a DHCP server (it doesn’t matter that you configured a Static IP Pool on the spreadsheet) and since there isn’t any, you get a 169.254.x.x IP and the validation fails. For example:

VM Kernel ping from IP '172.22.17.2' ('NSXT_EDGE_TEP') from host 'esxi1.vsphere.local' to IP '169.254.31.119' ('NSXT_HOST_OVERLAY') on host 'esxi2.vsphere.local' failed
You can see the IP is on the 169.254.x.x range

Luckily, this is just a validation bug, it is reported internally, and will likely be fixed in the latest VCF release. The issue will not present itself while actually doing the deployment and the TEP addresses will be set up correctly using the static IP Pool

BGP Route Distribution Failure

If your BGP neighboring is not configured correctly on your upstream routers, you will see the task “Verify BGP Route Distribution fail”

021-04-08T05:09:54.729+0000 [bringup,42ba3b72e2ee4185,395f] ERROR [c.v.v.c.f.p.n.p.a.VerifyBgpRouteDistributionNsxApiAction,pool-3-thread-13] FAILED_TO_VALIDATE_BGP_ROUTE_DISTRIBUTION
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to validate the BGP Route Distribution result for edge node with ID 123b3404-bab6-4013-a9f7-eba3b91b4faf

This means that the BGP configuration on the upstream routers is incorrect, usually, there is a BGP neighbor missing. The easiest way to figure out what’s missing is to check the BGP status on the Edge Nodes

In my case, the Upstream switches only had one neighbor configured per uplink VLAN, so node 1 showed:

BGP neighbor is 172.22.15.1, remote AS 65211, local AS 65210, external link
BGP version 4, remote router ID 172.22.15.1, local router ID 172.22.16.2
BGP state = Established, up for 09:09:51

And node 2 Showed:

BGP neighbor is 172.22.15.1, remote AS 65211, local AS 65210, external link
BGP version 4, remote router ID 0.0.0.0, local router ID 172.22.15.3
BGP state = Connect

You can see that the BGP session for node 2 is not established. After configuring the neighbor correctly on the upstream routers, the issue was resolved!

Conclusion

Deploying VCF 4.2 in this environment has been a rollercoaster but luckily, all the issues were able to be solved.

I hope this helps you either avoid all of these issues (by pre-emptively checking and fixing what could go wrong) or in case it does happen to you, to fix them as quick as possible)

Stay tuned for more VCF 4.2 adventures, next time, with workload domains!

How do I get to vSphere 7.0 without dying in the process?

Hello Everyone,

After a long hiatus, I decided to write a new blog post (and hopefully improve the frequency of them :D) – This will be based on a 2-hour presentation that I did for VMUG (VMware User Group) Argentina last week, which was done in spanish, and I will link it down below

However, for all of the non-spanish Speakers, I will do a breakdown of everything you need to check before attempting a vSphere upgrade from the vCenter & PSC perspective to pass the upgrade wth flying colors! – Buckle up!

Where is our environment currently standing?

First of all, you need to assess the current situation of your vCenters and PSCs – Is replication working correctly for example? This article goes really really deep into checking that:

Pre-upgrade considerations in Multi-vCenter environments

If you have any replication issues, this is the first thing you need to fix, otherwise, as shown in the previous article (and the video) you risk completely destroying your environment.

The 2nd thing you need to check is your current topology – How many PSCs and vCenters are actually in my environment? Am I using PSC HA? Is everything converged? Depending on your current topology, it might be a pretty trivial migration or it would need multiple steps over the course of a weekend.

What happens in the upgrade process?

First of all, the external PSC is deprecated in vSphere 7.0 – That means that, as a part of the upgrade process, any environment with an external PSC is converged. Even though this process might be straightforward, it can cause multiple problems before, during and after the migration. It’s easier and more convenient to break it up in parts

So if we’re good with replication (check and re-check previous article, I can’t stress this enough) then we need to figure out an upgrade and migration plan

Planning the upgrade process based on our topology

Let’s start with something simple:

What would be the correct steps here?

Let’s break it down:

1: Offline snapshots of all three VCs (with embedded PSCs) – offline means with all the SSO domain powered off- this is done from the ESXi nodes that are hosting the VMs.

2: Upgrade vCenter 1

3: Check functionality and replication

4: Offline snapshots of all three VCs (with embedded PSCs)

5: Upgrade vCenter 2

6: Check Functionality and replication

7: Guess what?

8: Upgrade vCenter 3:

9: Check Functionality and replication

10: Delete all snapshots

Why am I taking snapshots at every step? Why don’t I just take a single round of snapshots and then upgrade all at once?

Well, because if you had any issue at any point of the 2nd or 3rd upgrade, you would have to roll back everything and start from scratch. If you do it this way, you have multiple points to go back and avoid having to re-do the upgrade process! This can get even worse if instead of 3 vCenters you have 9 or 10 – If let’s say, you had an issue with upgrade 7, you would have to revert everything!

Now let’s make this a little bit more complicated!

So let’s picture this scenario (which is not too uncommon, i’ve seen this is in the real world)

What do we have?

First of all, blue lines symbolize good replication and red lines symbolize that replication is not working – So, as discussed earlier, this will be the first thing to fix – in the process of fixing this (most likely with a GSS ticket), multiple rounds of offline snapshots will be taken!

Now, onto the topology:

  • 6 External PSCs in a ring topology
  • 3 PSC HA VIPs being used by 2 vCenters each
  • 6 vCenters

So what should we do here? This not only involves the upgrade of the vSphere environment, but also, the re-pointing of 2nd and 3rd party tools to the new converged PSCs – Think of NSX and SRM for example.

The biggest pain point in this scenario, however, is PSC HA – how do we get rid of this prior to the upgrade?

Even though there is a KB for converging PSC HA (https://kb.vmware.com/s/article/65129) in practice, this is not the best approach due to how error prone it is.

What is the best approach? There are two ways to approach this, depending on downtime and operations.

The cleanest approach, would be to deploy 6 new PSCs, then repoint the vCenters to those 6 PSCs, and then decomission all the PSC HA nodes (as well as the VIP) – However, this might be complicated because of lack of IP addresses in the management segment, time, etc.

You could also leverage lsdoctor (https://kb.vmware.com/s/article/80469) to unconfigure PSC HA and then repoint the vCenters to each of the nodes – This introduces a little bit more downtime per vCenter (downtime when unconfiguring PSC HA + downtime until the repoint is complete) but removes the need of deploying new PSCs.

If you ask me, I recommend the first option, to make this as clean as possible.

So in this scenario, what would you do?

  1. Offline snapshots of all vCenters and PSCs
  2. Deploy PSC 7 pointed to PSC 6
  3. Deploy PSC N pointed to PSC N-1 until all PSCs are deployed.
  4. Check replication among the new PSCs

So now we have something like this

You can see that by deploying the PSCs in that order, we have a “semi-ring” already, with way less operational hassle than if we were deploying them pointed to a single PSC and then having to remake the replication agreements

So what’s next?

We need to repoint the vCenters to these new PSCs – Since the repoint is a pretty short process, you can get away with taking a single round of offline snapshots at the beginning and just repoint everything

  1. Offline snapshots of all vCenters and PSCs
  2. Repoint all vCenters to the new PSCs, 1:1
  3. Check correct functioning

End result:

Lovely, right?

Now, we need to get rid of all the PSCs that were forming the PSC HA (nodes and VIPs)

  1. Offline snapshots of all vCenters and PSCs
  2. Decomission all PSCs and PSC HA VIP nodes using: https://kb.vmware.com/s/article/2106736
  3. Check correct functioning

Now we’re here!

So we did all this and we haven’t even started upgrading or converging… but believe me, taking due diligence in doing this as clean as possible will save you from multiple headaches when you actually upgrade!

So what is left?

  1. Form a ring creating an agreement between PSC12 and PSC7
  2. Take a new round of offline snapshots
  3. Converge PSC7
  4. Check correct functioning
  5. Take a new round of offline snapshots
  6. Converge PSC8
  7. ….
  8. ….
  9. Until all PSCs are converged

In case there is any issue with the convergence, you can just go back to the latest functioning snapshot so you don’t have to redo everything!

You should be here now:

And from here, you can finally do the upgrade process – as discussed previously and in the first scenario, you should take a round of offline snapshots per each upgrade, to avoid having to re-do upgrades

Last but not least, you should repoint all 2nd and 3rd party solutions to the new converged (and upgraded) PSCs that are now living inside the vCenter appliance!

Closing note

I hope you enjoyed this post – If you have even limited knowledge of spanish, I encourage you to watch the youtube video in which I go over this in detail, and also I analyze and fix replication issues the same way it would be done if you contacted GSS.

Feel free to share this with peers, customers, partners – If we generate awareness about these processes and a clean and correct way of doing them, we will have way more succesful upgrades!

Quickly create NSX-T Segments using PowerCLI and NSX-T REST API!

Hello Everyone,

In today’s edition, I’m going to share with you a script that I wrote that will do the following:

  • Get all the VMs from your infrastructure
  • For each VM (and each virtual nic that is connected to a portgroup), it will query
    • Portgroup Name
    • Portgroup VLAN Type (because we’re going to skip trunk VLANs)
    • Portgroup VLAN ID
    • Default gateway (here is where it gets tricky…)
    • Gateway network prefix (also tricky)

Once we have all that data, we will proceed to create all of these segments (with the gateway and network prefix) inside NSX-T, using REST API calls.

By creating the segments with the gateway, we serve two great purposes

  • The segment is already prepared to be connected to a T1 DR and does not need further manual editing
  • Customers may not know all the gateways of all their vSphere networks, and this script will output that for you!

What do I need from you to run it?

  • vCenter & NSX Manager FQDNs and credentials
  • NSX-T Overlay transport zone name (the transport zone we’re going to use to create the segments)

This is the link to the script, where you can take a look at the code: https://github.com/luchodelorenzi/scripts/blob/master/createSegments.ps1

I’m going to explain it bit by bit, mostly focusing on the logic and the problems I encountered while testing it.

The end result would be something like this:

Enter vCenter FQDN: vcsa-01a.corp.local

PowerShell credential request
Enter vCenter Credentials
User: administrator@vsphere.local
Password for user administrator@vsphere.local: ********

Enter NSX Manager FQDN: nsxapp-01a.corp.local

PowerShell credential request
Enter NSX Credentials
User: admin
Password for user admin: ****************

Enter NSX Overlay Transport Zone name: nsx-overlay-transportzone

Name                           Port  User
----                           ----  ----
vcsa-01a.corp.local            443   VSPHERE.LOCAL\Administrator
Querying data for rdsh-01a ...
Querying data for log-01a ...
Querying data for web-01a ...
Querying data for vm-01a ...
Querying data for app-01a ...
Querying data for web-02a ...
Querying data for edgenode-01a ...
############################################################
Found the following possible segments in your infrastructure
Portgroup PG-WEB with VLAN 100 gateway 172.16.10.1 and prefix length 24
Portgroup PG-VM with VLAN 200 gateway 172.16.101.1 and prefix length 24
Portgroup PG-APP with VLAN 300 gateway 172.16.20.1 and prefix length 24
Would you like to Create these segments on NSX-T?
 ( y / n ) : y
Yes, create segments
found transport zone id: 1b3a2f36-bfd1-443e-a0f6-4de01abc963e
Creating Segment PG-WEB-VLAN100-GW-172.16.10.1 on transport zone nsx-overlay-transportzone
Creating Segment PG-VM-VLAN200-GW-172.16.101.1 on transport zone nsx-overlay-transportzone
Creating Segment PG-APP-VLAN300-GW-172.16.20.1 on transport zone nsx-overlay-transportzone
Simple, right?

So let’s start breaking the script up in parts…

Part 1 – Getting FQDNs and Credentials

This is pretty self explanatory, we’re just getting the FQDNs and credentials and saving them into variables.

$vcenter=Read-Host "Enter vCenter FQDN"
$vccredential = Get-Credential -message "Enter vCenter Credentials"
$nsxmanager=Read-Host "Enter NSX Manager FQDN"
$nsxcredential = Get-Credential -message "Enter NSX Credentials"
$overlayTransportZone = Read-Host "Enter NSX Overlay Transport Zone name"
Connect-VIServer -Server $vcenter -credential $vccredential

Part 2 – Exporting Data from VMs

So what are we doing here?

  • We’re iterating through every VM in the infrastructure, and getting the IP stack (which is part of the extensiondata.guest object, therefore, being read from VMware tools – This would be empty if this VM does not have VMware tools running)
  • We’re getting the device (virtual nic) that has the portgroup with the default gateway (this is needed in the scenario of multiple virtual nics and multiple portgroups)
    This is important because the VMware tools data is not available in vSphere, so we need to do the mapping ourselves.
  • We filter stuff out that we don’t need, such as:
    • Any network with less than 6 characters (ipv6 empty network)
    • Any network that does not have “.” on the address (so no ipv4)
    • Any prefix length that is 0 or 32 (useless in this scenario, this is not the gateway network)
    • Any network that starts with 224. or 169.254

After having filtered that, we’re going to have our gateway and network prefix, so what’s next?

  • Using the device, we get the portgroup that we’re going to use, and from that portgroup, we get the VLAN configuration and VLAN ID, and we discard it if it is a trunk portgroup
  • We will also discard the portgroup if it contains “vxw-dvs” because this will mean it is a NSX-V portgroup and won’t be VLAN backed
  • We create a new object that will contain:
    • Portgroup Name
    • Portgroup VLAN ID
    • Gateway
    • Network prefix
  • And we add this object to an array of objects
foreach ($vm in $vms) {
    $networkObject = "" | Select Portgroups,Gateway,Prefix
    $networkObject.Portgroups = ($vm | Get-NetworkAdapter | Get-VDPortgroup)
    Write-Host Querying data for $vm...
	if ($vm.extensiondata.guest.ipstack){
		$device = ($vm.extensiondata.guest.ipstack[0].iprouteconfig.iproute | where {$_.network -eq "0.0.0.0"}).gateway.device 
		$networkObject.gateway = ($vm.extensiondata.guest.ipstack[0].iprouteconfig.iproute | 
			where {$_.network -eq "0.0.0.0"}).gateway.ipaddress
		$networkObject.Prefix = ($vm.extensiondata.guest.ipstack[0].iprouteconfig.iproute | 
			where {$_.network.length -gt 6} | where {$_.network -like "*.*"} | 
				where {$_.prefixlength -ne 32} | where {$_.network.substring(0,4) -ne "224."}  | 
					where {$_.prefixlength -ne 0} | where {$_.network.substring(0,8) -ne "169.254."} | 
						where {$_.gateway.device -eq $device}).prefixlength
						
		if (($vm | Get-NetworkAdapter)[$device]){
			$pg = ($vm | Get-NetworkAdapter)[$device] | get-vdportgroup
		}
		$PGObject = "" | Select Name, VLAN, Gateway, PrefixLength
		$PGObject.Name = $pg.name
		$PGObject.VLAN = $pg.VlanConfiguration.VlanId
		$PGObject.Gateway = $networkObject.Gateway
		$PGObject.PrefixLength = $networkObject.Prefix
		#Skip Trunk vLAN
		if ($pg.VlanConfiguration.vlantype -ne 'Trunk' -and $pg.name -notlike "*vxw-dvs*" -and $pg.name -ne $null){
			$PossibleSegments += $PGObject
		 }
	}

Part 3 – Parsing the data

We have an array of objects that have all the data we need, but this will likely have many repeated entries, since a lot of VMs are going to be using the same portgroup and same gateway. We could use a single entry per portgroup, but this will not be ideal.
There is nothing stopping anyone from using multiple networks inside a same portgroup and VLAN, so the ‘uniqueness’ of the segment will be given by a combination of the portgroup as well as the gateway
In that way, we will have all the data we need and won’t discard anything useful!

$UniqueSegments = $PossibleSegments | Where {$_.Gateway -ne $null} | sort-object -Property  @{E="Name"; Descending=$True}, @{E="Gateway"; Descending=$True} -unique

Write-Host "############################################################"
Write-Host "Found the following possible segments in your infrastructure"
$uniqueSegments | % {
	Write-Host Portgroup $_.name with VLAN $_.VLAN, gateway $_.gateway and prefix length $_.prefixlength
}

Part 4 – Pushing the data to NSX-T

So now that we have our array of segment objects fully sorted out and having unique entries, we need to push it to NSX-T

Remember at the beginning I asked for the NSX-T Overlay Transport zone? We’re going to need the transport zone ID

NSX-T 3.0 Rest API – List transport zones
So with the name, we can execute that API call and get the transport zone ID, to use it in the create segment API call!
NSX-T 3.0 Rest API – Create Segment

$getTzUrl = "https://$nsxmanager/api/v1/transport-zones"
	$getTzRequest = Invoke-RestMethod -Uri $gettzurl -Authentication Basic -Credential $nsxcredential -Method get -ContentType "application/json" -SkipCertificateCheck
	$gettzrequest.results | % {
		if ($_.display_name -eq $overlayTransportZone){
			$overlayTzId = $_.id
			Write-Host found transport zone id: $overlayTzId
		}
	}
	foreach ($segment in $uniqueSegments)
	{
		$segmentDisplayName = $segment.name + "-VLAN" + $segment.VLAN + "-GW-" + $segment.gateway
		$Body = @{
			display_name = $segmentDisplayName
			subnets = @(
					@{
					gateway_address = $segment.gateway + "/" + $segment.prefixlength
					}
				)
			transport_zone_path="/infra/sites/default/enforcement-points/default/transport-zones/$overlayTzId"
				}	
		$jsonBody = ConvertTo-Json $Body
		Write-Host "Creating Segment $segmentDisplayName on transport zone $overlayTransportZone" 
		$patchSegmentUrl = "https://$nsxmanager/policy/api/v1/infra/segments/" + $segmentDisplayName
		$patchRequest = Invoke-RestMethod -Uri $patchSegmentUrl -Authentication Basic -Credential $nsxCredential -Method patch -body $jsonBody -ContentType "application/json" -SkipCertificateCheck
	}
}

Conclusion

I hope you find this script useful – It should GREATLY improve times in fresh NSX-T deployments, not only by quickly creating all the segments automatically, but also since it does all the hard work of exporting all the gateway and prefix configuration, which can be super tedious!

If you have any questions regarding the script, please leave it on the comments below and I’ll address it

If you found any bugs or errors or better ways to accomplish the same thing, please also leave a comment! It will also be super helpful!

Cheers,