Category: NSX-HomeLab

NSX-T Home lab Setup and BGP Configuration

In this blog post, I have covered, How I have configured BGP peering between NSX-T Edge and Cisco 1000v virtual router.

I have explained my lab environment in my previous blog post

I have already deployed following virtual components on Physical Server.
1. Three ESX Servers (version 7.0 U1)
2. One vCenter (version 7.0 U3)
3. One DNS Server (Cent-OS)
4. One NSX Manager (version 3.1)
5. One NSX Edge (Version 3.1)
6. One Cisco CSR Router
7. Linux based test VMs and Windows VMs (2016)

Physical Topology

NSX-T LAB Deployment

Logical Topology

In above logical topology, I have configured Windows VM with ip (192.168.10.100) and it’s default gateway is Cisco 1000v (192.168.10.2), Here is the configuration details for Cisco 1000v. Especially 2 interfaces, one is facing windows VM and other one is facing NSX-T Edge Gateway.

Interface configuration
Snippet of Interface Level Configuration

Ping works well between 1000v (192.168.10.2) and Windows VM (192.168.10.100)

Here is quick glimpse of vSphere Compute and Network configurations.

Deployed 3 nested ESXi servers and each ESXi has test VM (web-VM) in running state
DVS Configuration and VM-Tier-1/2/3 segments deployed by NSX-T

Here is the configuration specs for NSX-T. I have prepared following 3 ESXi hosts with NSX-T.
In NSX Configuration column is showing success status for each ESXi host. It means NSX vibs are successfully installed in ESXi hosts

I have selected labesx-1.vnetworkcloud.com, Click on Action menu to check overlay configuration.

Type :- NSX-VDS
Mode :- Standard
Name :- NSX-VDS
Transport Zone :- nsx-overlay-transportzone (it covers 3 ESXi & 1 Edge Gateway)
Uplink Profile :- ESX-hostuplink-profile
TEP IP Segment :- Static IP Pool (192.168.1.80-85)

Single NSX-T Edge Service gateway prepared with NSX vibs and TEP IP Configured on it. (I have not configured edge cluster because of limited underline resources)

I have selected nsx-edge1, Click on Action menu to check overlay configuration.

NSX-T Edge Configuration specs

Until now I have created 2 transport zones, overlay back TZ spans on 3 ESXi and Edge-1. VLAN back TZ spans only on Edge-1.

ESXi uplink profile configuration specs

Overlay and VLAN Back Configuration specs.

I just have single vCenter deployed in this environment and i have added that vCenter into the compute section of NSX-T.

From here, I have deployed NSX-T segments followed with Tier0/1 Gateway deployment.
3 logical segments (VM-Tier-1/2/3) are overlay backed are connected to tier1 router and 1 Edge-uplink1 is VLAN uplink for Edge-1

Tier-0 GW deployed

Configuration of Edge Interface from Interfaces section.

We have configured Edge uplink interface 100.100.100.1/29 facing Cisco 1000v Router.
Next few screen-shot of BGP Configuration on Tier-0 GW.

BGP Enabled on Tier-0

Once BGP neighbors are configured, we need to redistribute the connected tier-0 segments

BGP Configuration and redistribution is completed on tier-0.
Now I have configured BGP on Nexus 1000v.

From here, I have deployed tier-1 gw and hooked tier-1 gw to tier-0 gw

This Tier-1 gw connected networks should be advertise if you want to communicate overlay segments with outside of NSX Network.

I have completed Tier0/1 and 1000v Configuration Part. Let’s explore the NSX CLI.
NSX command line verification on edge, Login to NSX-Edge (192.168.1.140) and get the logical routers details by running below command.

Command to list learnt BGP routes in NSX-T Edge.

Now I have checked Cisco 1000v route table. I can see routes learnt from NSX Segments

Ping connectivity test from Windows VM (192.168.10.100) to segments behind tier-1 gw

Here I have completed my second blog post on NSX-T Configuration in home LAB.
Next blog I will post NSX-T troubleshooting.

NSX Series 8 : ESX Server’s started communicating to NSX Controller

Yes, I have mentioned in my last blog. ESX hosts and NSX Controller communication couldn’t able to established between XYZ reason. after loads of troubleshooting i have decided to down grade my setup from NSX 6.2.5 to 6.2.0.

I have deleted NSX Manager 6.2.5 and deployed NSX Manager 6.2.0.

   1. Deployed NSX Manager 6.2, configured ip settings/NTP.                                                    2. Registered NSX Manager with VC Server.                                                                              3. Deployed NSX Controller-1, once NSX Controller deployment is completed.
4. Installled VXLAN vib on compute-ESX host and rebooted comp-ESX-1 host.
5. Reboot of Comp-ESX-1 host is completed, i restarted netcpad service to see ESX connection with NSX Manager :

[root@comp-esx-1:~] esxcli network ip connection list | grep -i 1234
tcp 0 0 192.168.20.20:61645 192.168.20.30:1234 ESTABLISHED 36846 newreno netcpa-worker

6. NSX Manager UI was showing channel health up and running..

channel-health-6.2

7. Controller has reported similar healthy status :

nsx-controller # show network connections of-type tcp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1234 0.0.0.0:* LISTEN 1808/java
tcp 0 0 127.0.0.1:1234 127.0.0.1:46173 ESTABLISHED 1808/java
tcp 0 0 192.168.20.30:1234 192.168.20.20:26114 ESTABLISHED 1808/java
tcp 0 0 192.168.20.30:1234 192.168.20.11:42080 ESTABLISHED 1808/java

8. configuration file /etc/vmware/config_by_vsm.xml,

[root@mgmt-esx-2:~] cat /etc/vmware/netcpa/config-by-vsm.xml
<config>
<connectionList>
<connection id=”0000″>
<port>1234</port>
<server>192.168.20.30</server>
<sslEnabled>true</sslEnabled>
<thumbprint>F6:61:B4:A1:8D:A9:D2:6F:8C:E7:91:96:52:E2:90:D7:57:72:99:B0</thumbprint>
</connection>
</connectionList>
<localeId>
<id>564D9510-63AA-8EC4-3BAB-6A575B24AD96</id>
</localeId>
<vdrDvsList>
<vdrDvs id=”0000″>
<numActiveUplink>1</numActiveUplink>
<numUplink>4</numUplink>
<teamingPolicy>FAILOVER_ORDER</teamingPolicy>
<uplinkPortNames>PhysicalUplink,ManagementNetwork,vtepNetwork,vMotionNetwork</uplinkPortNames>
<uuid>6c f1 14 50 23 a5 8e a3-5d 14 bf 22 1a d6 8b a1</uuid>
<vxlanOnly>true</vxlanOnly>
</vdrDvs>
</vdrDvsList>

9. netcpa certificate files was showing some valid keys into it.

[root@mgmt-esx-2:~] cat /etc/vmware/ssl/rui-for-netcpa.key
—–BEGIN RSA PRIVATE KEY—–
MIIEowIBAAKCAQEAyDNB5tcDdAfwAQy3OTDr4HOfYAifR4R6Cp5i6r5GXOxYVVlp
5B+jpj6kmGimFaLvRRTuPp8DLEp2kPLdKXtS7Voepv0xcRP03/ikhzXOdIJCS6bo
4s+2G0OG66GTRViNWg2+WNchJlLQSNDBuvk8KrJ128ea7kEaLObWMPeffPYbbrHD
dD2fe+3a7JionQkWvVmKn2ryxdsmpJhnFoEygdNnc/aPdHFIayZcpsw/yDXeyul8
fsBcJeRsWgBYe8Y9N64Kn2TvW7ZJnJ7+yD3hZ1ijk9zRcokcasfblAL47SOdWmb9
gRqFj8AbzYSY2XduvwVJKdoI3NEngl/awfU+OQIDAQABAoIBAQC3VblUHDMdXud7
Dpn71cEsmrWozDyQ0dCxSLptHjJxhj6eanN+ZTsIUvWIXM07oypaqQBEeHn/ZXGO
69l802Lt2UmYwuU0GIJ3WMvSxoN6To9HmGJM0sqBzSURQ7YKGz3W6h8WXGwPHGeB
RysifnuK3kLwMaq/5P6DgCFGVn3HgDqZ2VJs1a+b7p4PHP34pzTR2UJWWxVTYTNk
a/eILnTV0POdTcZDFPUaacD4MejX91iYPfNJzpf4+796VK3+ceibbwBQyCGR+CgB
DQD/OR1hF6Kkr4IhmkXTThy0QDb+N3rQQ5GlsHKjv/+7Emv65UVQGb4yDu0JJxVS
OS147VfBAoGBAOOtIxAT4vpT2LODVgbbQbYc0x6HD/NHD7G7X5h7BS8guFlXkO2z
PB6d65LXgr4GpoTBf/kN8kcgd9d1BJ31uCRLObD9fXSb4q0K10GwQsYCPLUQD3UJ
J2KGbIeLngyAd9I+516atdLRFNPaVWtI4XM1Os2E9UOUdnCsne1lLQqzAoGBAOEb
FSqDDtYVdVd3wARInZ7DRZF/N4b5nTk85bKbmGrTZa3AZGNfOSewTBMrIyWUGw3O
E4oYjT0M+94H6Kba0fC8lHHuCJHpJMjkRXGPCAU9KZDB1LPmxP74RLzrrL2/zeEM
5naGkEYB0HYs2FX6tPdroRA9KxIHF1wGW1neVvljAoGANqHfHgaHmqXwA/Np+y81
r2tvRY4HR8sveuiJYd9zOQHS+q1Ge6Ncs8QVOSE5oDucTZLZEHxGPMcPusW7m+PQ
kfTzHQNWzVpypC9Lw9l1zGnRA2mdK+pMZxn2hPf9ruZM2kh54HujUbkWFFf500qj
vq9xMsvKxrKjURo6szJqAn0CgYAukMCnI6dNB/FXH7F78V2RuwlIGM3uwPISJhPK
B7/W0VYmRDdACzGykRibmn/RxTp3d5q5JW0KJQZxHqe05G0rZIc8riXdsSHn4Liw
zgq1srpHdKFAectkhw0JG6OHOQcot9dwy1NK+BMcbxhtaMhIql4OEuk2+HqPkc0R
4RmSywKBgCj7cR6SzELyHA0MmerEV6pWH8fS83khedVe8s7pezkOzGKy0sRCTeGG
D4LsXHwP4m0LDUrEZZHQnQpTtBvwTW+Pq8CRpa7O2P8IqoTl4CxWzct2Yxl0vqZ9
nA7HqA3UTKMPx6718qDWq+SnO61EAQKOC44RBwcz/THa3uzeOTrt
—–END RSA PRIVATE KEY—–
[root@mgmt-esx-2:~] cat /etc/vmware/ssl/rui-for-netcpa.crt
—–BEGIN CERTIFICATE—–
MIIDJDCCAgygAwIBAgIGAVyRr6wbMA0GCSqGSIb3DQEBCwUAMFMxUTBPBgNVBAMT
SFZNV2FyZSBWWExBTiBIb3N0IENlcnRpZmljYXRlIGhvc3QtODEgT1U9TmVjdHdv
cmtpbmcgTz1WTVdhcmUgU1Q9Q0EgQz1VUzAeFw0xNzA2MTAxMTA3MDhaFw00NDEw
MjYxMTA3MDhaMFMxUTBPBgNVBAMTSFZNV2FyZSBWWExBTiBIb3N0IENlcnRpZmlj
YXRlIGhvc3QtODEgT1U9TmVjdHdvcmtpbmcgTz1WTVdhcmUgU1Q9Q0EgQz1VUzCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMgzQebXA3QH8AEMtzkw6+Bz
n2AIn0eEegqeYuq+RlzsWFVZaeQfo6Y+pJhophWi70UU7j6fAyxKdpDy3Sl7Uu1a
Hqb9MXET9N/4pIc1znSCQkum6OLPthtDhuuhk0VYjVoNvljXISZS0EjQwbr5PCqy
ddvHmu5BGizm1jD3n3z2G26xw3Q9n3vt2uyYqJ0JFr1Zip9q8sXbJqSYZxaBMoHT
Z3P2j3RxSGsmXKbMP8g13srpfH7AXCXkbFoAWHvGPTeuCp9k71u2SZye/sg94WdY
o5Pc0XKJHGrH25QC+O0jnVpm/YEahY/AG82EmNl3br8FSSnaCNzRJ4Jf2sH1PjkC
AwEAATANBgkqhkiG9w0BAQsFAAOCAQEAKJRVUIQt8gRyNlXsd10AojLNiUyZqXP5
NsDgEo9BB3qviAY0s/6SnoxRch9O23krHvJH0x8R+TX0h7mfwdqxAWTC7+RYjH1f
1EpTuazXZ0k38e7A+yE0BZNl66n5xNm4AJ63NgvgKctYETXn89VJhf/4kzm6hX3c
pKm5UEQgxqZkWQNqthqj3Xz3US5VtGab53oTgiCeZAoSIqrpQ+OuAQWx7x4ezJ14
ud4hufsIUFNSYxtuhSaeHen2rdexD7hLTdKv/tvUJLBBzZG/mi17+343LISvk3Bh
VcMhixNQhWOquhCUvuaFjexyQov8BPA0JfYad9kN8ikTxgom8n1jww==
—–END CERTIFICATE—–
[root@mgmt-esx-2:~]

NSX Manger ova 6.2.5, the one i have download has some issue because i just changed the NSX Manger ova to 6.2 and everything started working as expected.

 

 

NSX Series 7 : ESX Server’s unable to communicate to NSX Controller

I had almost spent a week of time to troubleshoot this issue but i couldn’t able to get the answer why ESX host unable to communicate to NSX Controller “Just one controller deployed in my lab”.. I just note down steps i took to troubleshoot this issue

  1. DNS deployed could able to resolve it.
  2. Time was sync on all the ESX/VC/NSX Manager 6.2.5
  3. NSX Manager/VC Server/DNS/ all are belongs to same network : 192.168.20.X
  4. netcpad restart multiple times.

Here is detail step i followed to troubleshoot this issue but couldn’t able to solve it 😦

  1. DNS forward and reverse lookup is working as expected.. i tried the command from

  2. VC Server “Linux appliance”vc-server:~ # nslookup nsx-manager
    Server: 192.168.20.2
    Address: 192.168.20.2#53

    Name: nsx-manager.vnetworkcloud.com
    Address: 192.168.20.4

    vc-server:~ # nslookup 192.168.20.4
    Server: 192.168.20.2
    Address: 192.168.20.2#53

    4.20.168.192.in-addr.arpa name = nsx-manager.vnetworkcloud.com.

     

2. NSX Manager and VC Server lookup service status was overall in green status. As time was sync in between NSX/VC and DNS worked well hence i couldn’t get any to register NSX Manager with VC Server.

Lookup.PNG

      3. Here is real fun starts. I have deployed one controller in my home lab and installed VXLAN vibs on management and compute cluster.  I have rebooted compute/Management ESX hosts once the vxlan/vsip vibs were installed.

hostpre

      4.  First command i tried to check the connection between ESX host and NSX controller is,

[root@mgmt-esx-2:~] esxcli network ip connection list | grep 1234

[root@mgmt-esx-2:~]

I was expecting some output from the above command but it didn’t return anything hence i logging to controller n make sure that controller is listening on port 1234

nsx-controller # show network connections of-type tcp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1234 0.0.0.0:* LISTEN 2259/java
tcp 0 0 127.0.0.1:2003 0.0.0.0:* LISTEN 1489/python
tcp 0 0 127.0.0.1:2004 0.0.0.0:* LISTEN 1489/python
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2100/sshd
tcp 0 0 127.0.0.1:9080 0.0.0.0:* LISTEN 2287/domain
tcp 0 0 127.0.0.1:7002 0.0.0.0:* LISTEN 1489/python

5. I restarted netcpad multiple times but no luck. /var/log/netcpad.log file was showing some ssl related alerts hence i tried searching netcpad configuration file and related ssl file

017-06-10T06:51:41.350Z info netcpa[FFA14F00] [Originator@6876 sub=Default] VSM configured privateKeyFile /etc/vmware/ssl/rui-for-netcpa.key, certificate /etc/vmware/ssl/rui-for-netcpa.crt

2017-06-10T06:51:41.351Z info netcpa[FFC1DB70] [Originator@6876 sub=ThreadPool] Thread enlisted

2017-06-10T06:51:41.351Z info netcpa[FFC5EB70] [Originator@6876 sub=ThreadPool] Thread enlisted

2017-06-10T06:51:41.351Z info netcpa[FFC9FB70] [Originator@6876 sub=ThreadPool] Thread enlisted

2017-06-10T06:51:41.351Z info netcpa[FFCE0B70] [Originator@6876 sub=ThreadPool] Thread enlisted

2017-06-10T06:51:41.352Z info netcpa[FFD21B70] [Originator@6876 sub=ThreadPool] Thread enlisted

2017-06-10T06:51:41.352Z info netcpa[FFD62B70] [Originator@6876 sub=ThreadPool] Thread enlisted

2017-06-10T06:51:41.354Z info netcpa[FFA14F00] [Originator@6876 sub=Default] Vmacore::InitSSL: handshakeTimeoutUs = 20000000

2017-06-10T06:51:41.355Z error netcpa[FFA14F00] [Originator@6876 sub=Default] Failed to initialize the SSL context: Crypto Exception: error:02001002:system library:fopen:No such file or directory

      6. Here is the content of config_by_vsm.xml file

[root@mgmt-esx-2:/etc/vmware/netcpa] cat config-by-vsm.xml
<config>
<!– this is an example for how to configure controller list
<connectionList>
<connection id=”0″>
<server>10.35.195.185</server>  ===> NO idea from where it took this ip
<port>1234</port>
<thumbprint>16:86:44:85:D7:A8:AB:3D:19:EB:D9:B0:75:F4:4C:50:26:C5:E8:66</thumbprint>
<sslEnabled>false</sslEnabled>
</connection>
</connectionList>
–>
<!– this is an example for how to configure vdr instance list
<vdrInstanceList>
<vdrInstance id=”0″>
<vdrId>1</vdrId>
<vdrName>vdr1</vdrName>
</vdrInstance>
</vdrInstanceList>
–>
</config>

       And ssl certificates files was blanks. Actually it should have some certificate information.

[root@mgmt-esx-2:/etc/vmware/netcpa] cat /etc/vmware/ssl/rui-for-netcpa.crt

#

# Empty ssl crt file for persisting.

#

[root@mgmt-esx-2:/etc/vmware/netcpa] cat /etc/vmware/ssl/rui-for-netcpa.key

#

# Empty ssl key file for persisting.

#

    7. NSX Manger shows some negative result when i ran this command :

nsx-manager> show host host-81 health-status

UNHEALTHY, Got exception when checking host host-81 health status: Host host-81returns no data for command “esxcli system module list”

Status: UNHEALTHY

channel.png

Then i came across one blog which reported similar issue which i have mentioned here that guy has cloned the NSX Manager VM but in my lab i have not cloned the NSX manager. Don’t know what cause this issue.. anyways i will keep this post open and will update this post once i have got any answer.