NetApp 07 - Access and directory integration

NetApp Solution · Previous: SVMs and NFS · Next: Encryption and certificates

A MetroCluster site is not one device to log in to but a dozen: two clusters, four nodes with their service processors, four Fibre Channel switches, four bridges and the management software. The requirement was simple, personal accounts from Active Directory everywhere and a local account only as the last resort, and each family of devices met it differently: ONTAP after a detour that lasted from September into November 2017, the Brocade switches through a different protocol, the ATTO bridges not at all.

Roles, groups and accounts

There are two roles. Administrators have full rights on every component. Operators have read-only rights; the same role is used by monitoring.

In Active Directory the design separates role groups, which the devices know, from authorization groups, which hold the people. The role group NETAPP_ADM contains the group NETAPP_ADM_ORG with the organisation's administrators; the operator role is built the same way. User accounts live in OU=Users_STD, the groups under OU=RBAC, and a service account netapp_svc in OU=Users_svc exists for LDAP queries.

The names of the operator groups are not consistent in the Source material: the design calls the role NETAPP_RO, with a member group written once as NETAPP_RO_ORG and once as NETAPP_OPR_ORG; the commands that were run use netapp_opr, and the notes on the switches netapp_ops. The commands are quoted below as they were run.

Local accounts remained on every component, for the day the directory is not reachable:

Two AD test accounts, netapp01 in the administrator role and netapp02 in the operator role, were used to prove each integration. A local account ocum was once created on the clusters for Unified Manager and is marked "not used" in the inventory. On the switches, root is disabled by default and reachable from the console only; the notes hold the three commands that enable it for remote login (userconfig --change root -e yes, rootaccess --set all, and passwdcfg --set -oldpasswd), without saying on which switches they were finally run.

Where to log in

Datacenter B and site 2 follow the same pattern with dc1-b-, dc2-a- and dc2-b-. The bridges speak neither HTTPS nor any AAA protocol; both facts are recorded as constraints C2 to C4 of the design and are the reason their management ports sit in the out-of-band network only.

flowchart LR
  adm["Administrator or operator"]
  subgraph ont["ONTAP, one cluster"]
    lif["Cluster management LIF, SSH and HTTPS"]
    asvm["Admin SVM, security login table"]
    tun["CIFS SVM DC1-S-VCVSM003, domain tunnel"]
  end
  subgraph fc["Fibre Channel"]
    sw["Brocade switch, SSH"]
    br["ATTO bridge, HTTP, local accounts only"]
  end
  tac["TACACS+ servers"]
  ocum["Unified Manager"]
  ad["Active Directory domain controllers"]
  adm --> lif
  lif --> asvm
  asvm -->|"domain account"| tun
  tun -->|"Kerberos or NTLM"| ad
  adm --> sw
  sw -->|"TACACS+, PAP"| tac
  tac --> ad
  adm --> br
  adm --> ocum
  ocum -->|"LDAP over TLS"| ad

ONTAP: the detour through LDAP

ONTAP offers two ways to let directory users administer a cluster. With the authentication method nsswitch the admin SVM treats the directory as a UNIX name service. With the method domain it hands the login to a CIFS server that is a member of the domain. The first looked like the direct way, a storage system that speaks LDAP to a directory, and took most of September 2017.

An LDAP client was created on the admin SVM of DC1-A-XNAS001, pointing at two domain controllers and binding with an already existing service account, vmail_svc, and the name service switch was told to use it:

$ ldap client create -client-config DC1-A-XNAS001_ldap -vserver DC1-A-XNAS001 -servers 10.11.16.209,10.11.16.210 -port 636 -query-timeout 10 -min-bind-level sasl -base-dn "DC=ad,DC=example,DC=net" -base-scope subtree -use-start-tls true -session-security none -bind-dn "cn=vmail_svc,ou=users_svc,DC=ad,DC=example,DC=net"
$ vserver services name-service ldap create -vserver DC1-A-XNAS001 -client-config DC1-A-XNAS001_ldap -client-enabled true
$ vserver services name-service ns-switch modify -vserver DC1-A-XNAS001 -database group,passwd -sources files,ldap
$ security login create -user-or-group-name SCB_ADM -application ssh -authentication-method nsswitch -role admin -vserver DC1-A-XNAS001 -is-ns-switch-group yes

The first line already contains its own contradiction: port 636 is LDAP over TLS from the first byte, StartTLS is an upgrade of a plain connection on port 389, and the two cannot be combined. What followed is preserved as a list of combinations, each ending with the same test.

$ diag secd connections test -node DC1-A-ANAS001 -vserver DC1-A-XNAS001

The notes do not record the port for the first five rows; it is inferred from the creation command and from the sixth attempt, which sets 636 explicitly. Only the last row, plain LDAP without any protection, connected. To take ONTAP out of the equation the same bind was tried from a Linux host with ldapsearch, plain and with -ZZ for StartTLS, and the traffic was captured with tcpdump on ports 389 and 636. The certificate of the domain controller was fetched with openssl s_client -connect 10.11.16.209:636 and installed on the cluster as a trusted CA:

$ security certificate install -vserver DC1-A-XNAS001 -type server-ca

The notes do not show an encrypted variant working afterwards.

With the connection up, the second problem appeared. The nsswitch method needs every administrator to be a UNIX user in the directory. The schema was copied and bent towards plain AD attributes (sAMAccountName as the user name, memberOf for groups), and the lookups were tested one by one at the diagnostic privilege level:

$ set -privilege diagnostic
$ diag secd authentication translate -node DC1-A-ANAS001 -vserver DC1-A-XNAS001 -unix-user-name admin01
$ diag secd authentication translate -node DC1-A-ANAS001 -vserver DC1-A-XNAS001 -uid 1
$ getXXbyyy getpwbyname -node DC1-A-ANAS001 -vserver DC1-A-XNAS001 -username admin01 -show-source true

Under the heading "Strange thing" the notes show where that leads: admin01 resolved to uid 1, and uid 1, which had been the local user daemon a moment before, now resolved to admin01. An AD without UNIX attributes has no sensible uidNumber to offer. The page of NetApp's TR-4073 copied into the notes lists what each account would need (uid, uidNumber, gidNumber, unixHomeDirectory, loginShell), next to a Microsoft article saying that Identity Management for UNIX is deprecated from Windows Server 2016 on.

The password was the third problem. The schema attribute for the user password was switched between userPassword and unicodePwd, with the results "connection OK, authentication fails" and "connection fails, authentication fails". The collected links explain it: Active Directory does not treat userPassword as a password unless a domain-wide heuristic is changed, and unicodePwd can be written but never read.

On 16 November 2017 the state was captured for a support case: every secd cache on both nodes cleared, the group cache timeout set to 0, a packet trace started against both domain controllers with pktt, and then:

$ secd authentication show-ontap-admin-unix-creds -node DC1-A-ANAS001 -vserver DC1-A-XNAS001 -unix-user-name admin01

Error: Acquire ontap admin UNIX credentials procedure failed
  [  0 ms] Entry for user-name: admin01 not found in the current
           source: FILES. Ignoring and trying next available source
  [  3002] Failed to initiate Kerberos authentication. Trying NTLM.
  [  5002] TCP connection to ip 10.11.16.209, port 389 via
           interface 10.11.10.37 failed: Operation timed out.
  [  5004] Successfully connected to ip 10.11.16.210, port 389
           using TCP
**[  5136] FAILURE: User 'admin01' not found in UNIX authorization
**         source LDAP.

This output is the most useful thing the whole detour produced. It shows which interface ONTAP uses for the query (the node management LIF 10.11.10.37, not the cluster management LIF, which matters for firewall rules), that one of the two domain controllers was not reachable from it on port 389, and that the user simply does not exist as a UNIX user.

ONTAP: the domain tunnel

The method that stayed asks nothing of the directory except what it is: a Windows domain. A data SVM with a CIFS server is joined to the domain, and the admin SVM is told to pass domain logins through it.

It had been tried in September too, through the SCB SVM DC1-S-VCVSM001, which was a domain member at the time for other reasons (see SVMs and NFS), and with an AD group named SCB_ADM as the test group. The final build uses two dedicated SVMs, one per cluster, because a domain tunnel is a property of a cluster and the two clusters of a MetroCluster are administered separately: DC1-S-VCVSM003 for DC1-A-XNAS001 and DC1-S-VCVSM004 for DC1-B-XNAS001.

Steps 5 and 6 on cluster A:

$ security login domain-tunnel create -vserver DC1-S-VCVSM003
$ security login create -user-or-group-name example\netapp_adm -application ssh -authentication-method domain -role admin -vserver DC1-A-XNAS001
$ security login create -user-or-group-name example\netapp_opr -application ssh -authentication-method domain -role readonly -vserver DC1-A-XNAS001

The same two security login create lines were repeated for http and ontapi, six entries per cluster. Today's documentation supports AD group access only for ssh, ontapi and rest; the group entries for http are outside that. The full command set is a Config document: ONTAP: CIFS server, LDAP client and domain tunnel.

Step 1 came from a community thread with the title "CIFS not joining AD domain"; the notes give no more reason than that reference. The LDAP client of step 3 is the remainder of the September work in a form that functions, because with -bind-as-cifs-server true the SVM binds with its own machine account over SASL and needs no service account and no password. It is enabled on both tunnel SVMs in the as-built report. The login itself, with the method domain, does not depend on UNIX attributes, which is why it works against this directory. The as-built report also shows a host entry for the first domain controller on both SVMs and a preferred domain controller on DC1-S-VCVSM004.

Two properties follow from the construction. The tunnel SVM is a data SVM and is mirrored like any other: after a switchover DC1-S-VCVSM004-mc starts on cluster A, with its LIF and its computer account. And an administrator's login now depends on the tunnel SVM, its LIF, DNS, time synchronisation and a reachable domain controller; the local admin account is the answer to all of these at once and its password must be where it can be found.

Site 2 was built when its own domain controllers did not exist yet. DC2-S-VCVSM003 and DC2-S-VCVSM004 were joined to the same domain through the domain controllers of site 1, and the design notes that they have to be reconfigured once site 2 has its own directory service.

Brocade: LDAP that could only bind anonymously

Fabric OS 8 has an LDAP client meant for Active Directory. It was configured on a switch as the manual says:

$ aaaconfig --add 10.11.16.209 -conf ldap -p 389 -d ad.example.net
$ aaaconfig --add 10.11.16.210 -conf ldap -p 389 -d ad.example.net
$ ldapcfg --maprole netapp_adm Admin
$ ldapcfg --maprole netapp_ops User
$ aaaconfig --authspec "ldap;local" -nologout

The role mapping showed up correctly in ldapcfg --show. The notes do not show the failed login itself; the design records the outcome as a constraint: the switches support only anonymous binding, and the domain controllers of the organisation do not allow anonymous binds. There is nowhere in aaaconfig to put a bind account. The notes end with a list of forum threads from other people locked out after switching to LDAP, and with the way back:

$ aaaconfig --authspec "local" -nologout
$ aaaconfig --remove 10.11.16.209 -conf ldap

The option -nologout matters on a device with one management path: it changes the authentication order without terminating the session in which the change is made.

Brocade: TACACS+

The way out was a protocol the switches implement fully. Two TACACS+ servers in the management infrastructure stand between the switches and the directory, and the switches were pointed at them, with the local database as the second source:

$ aaaconfig --add 10.11.17.17 -conf tacacs+ -s <TACACS_SHARED_SECRET> -a pap
$ aaaconfig --add 10.11.17.19 -conf tacacs+ -s <TACACS_SHARED_SECRET> -a pap
$ aaaconfig --authspec "tacacs+;local" -nologout
$ aaaconfig --show

TACACS+ CONFIGURATIONS
=====================

Position     : 1
Server       : 10.11.17.17
Port         : 49
Secret       : <TACACS_SHARED_SECRET>
Timeout(s)   : 3
Auth-Protocol: PAP

Position     : 2
Server       : 10.11.17.19
Port         : 49
Secret       : <TACACS_SHARED_SECRET>
Timeout(s)   : 3
Auth-Protocol: PAP

Primary AAA Service: TACACS+
Secondary AAA Service: Switch database

The command set is a Config document: Brocade: TACACS+ authentication. What the Source material does not contain is the other half: the configuration on the TACACS+ servers that maps an AD group to a Fabric OS role. Without a role attribute returned by the server a user would not get the intended rights, so such a mapping must have existed; it is not in the notes.

Bridges and management software

The ATTO FibreBridge 7500N has local accounts and nothing else: no LDAP, no RADIUS, no TACACS+, and its web interface is HTTP only. The compensating measures are the ones available: the management ports are in the out-of-band network, and the accounts are in the inventory of local accounts.

OnCommand Unified Manager talks to Active Directory directly over LDAP with TLS, both for its web interface and for SSH to its operating system. That configuration is in Unified Manager and API Services.

Lessons

• In the ONTAP of that time, "integrate with Active Directory" meant the domain tunnel; since ONTAP 9.16.1 the admin SVM can be joined to the domain itself. The LDAP name service is for file access by UNIX identities; using it for administrators requires a directory with UNIX attributes, and weeks went into finding that out.
• diag secd connections test and secd authentication show-ontap-admin-unix-creds say in one screen what a week of changing options does not: which interface, which server, which step failed. They should have been the first commands, not the last.
• The first LDAP client was created with port 636 and StartTLS together. Each of them is a valid choice; together they cannot work, and every later test inherited the confusion.
• A test with a service account and a group that belong to something else leaves entries behind. The login entries for the SCB group on the admin SVM were created three times with three spellings of the domain; the notes do not show them being deleted.
• A device that can only bind anonymously is not "LDAP capable" for a directory that is run properly. The question to ask a vendor is not whether LDAP is supported but how the device binds.