NetApp 10 - Unified Manager and API Services

NetApp Solution · Previous: Logging, monitoring and AutoSupport · Next: MetroCluster switchover and Tiebreaker

Two pieces of vendor software ran beside the storage on RHEL 7 virtual machines: OnCommand Unified Manager, the operations view of both clusters, and OnCommand API Services, a REST front end meant for the monitoring system. This article covers what they were for, how Unified Manager 9.4 was installed on a hardened RHEL 7.5, and why API Services ended up in a systemd-nspawn container on a virtual machine of its own, including the two things that did not work.

What they were for

The design gives each product one sentence. Unified Manager (OCUM) "is used for monitoring purposes and mainly for usage by operation team": one web interface that collects both clusters of a site, keeps history in its own MySQL database, raises events and sends mail. API Services "will be used for simpler (RESTful) integration of Netapp solution with central monitoring solution", Sensu. The clusters themselves were managed through ONTAPI, XML over HTTPS. API Services put a REST layer in front of that, and it took its inventory not from the clusters directly but from Unified Manager's database.

flowchart LR
  adm["Administrators and operators"]
  sensu["Sensu"]
  subgraph m1["DC1-A-VCOCM001"]
    ocum["Unified Manager 9.4"]
    db1[("MySQL 3306")]
  end
  subgraph m2["DC1-A-VCOCM002"]
    subgraph ct["nspawn container"]
      api["API Services 2.0"]
      db2[("MySQL 3306")]
    end
  end
  subgraph st["MetroCluster, site 1"]
    ca["DC1-A-XNAS001"]
    cb["DC1-B-XNAS001"]
  end
  ad["Active Directory"]
  mail["Mail server"]
  prx["HTTP proxy"]
  adm -- "HTTPS 443" --> ocum
  adm -- "HTTPS 8443" --> api
  sensu -- "REST 8443" --> api
  sensu -- "REST 443" --> ocum
  ocum --- db1
  api --- db2
  api -- "database user" --> db1
  ocum -- "HTTPS 443" --> ca
  ocum -- "HTTPS 443" --> cb
  ocum -- "LDAPS 636" --> ad
  ocum -- "SMTPS 465" --> mail
  ocum -- "AutoSupport" --> prx

Both sit in the in-band management VLAN 1024. Site 2 has its own DC2-A-VCOCM001; the notes hold no separate installation log for it.

The virtual machine

The whole data disk became one partition, one volume group vg01 and one 99 GB logical volume lv_data with ext4, mounted at /opt/netapp, which is where the product puts its binaries, its database and its installation packages.

Prerequisites: the product against the baseline

Unified Manager's installer assumes a plain RHEL. The baseline was not plain, and three of the preparation steps are the baseline being moved out of the way.

One line in the notes is fenced with exclamation marks: the password of umadmin must be admin at installation time, otherwise the installation fails. It is changed in the first-run dialogue of the web interface.

Installation

The package is a ZIP of six RPMs. It was unpacked under /opt/netapp/install/9.4/ (one directory per version, for the upgrades to come), the vendor's check script was run and its log read, and then everything was installed in one transaction.

$ cd /opt/netapp/install/9.4
$ unzip ./OnCommandUnifiedManager-rhel7-9.4.zip
$ ./pre_install_check.sh
$ less ./pre_install_check.log
$ yum install *.rpm

Installing:
 netapp-application-server          x86_64          9.4.0-2018.05.J158
 netapp-ocum                        x86_64          9.4-1806101600
 ocie-au                            x86_64          9.4.0-2018.05.J279
 ocie-server                        x86_64          9.4.0-2018.05.J279
 ocie-serverbase                    x86_64          9.4.0-2018.05.J158
Installing for dependencies:
 mysql-community-server             x86_64          5.7.23-1.el7
...
OnCommand Unified Manager installed successfully.

The listing is abridged; the sixth product package, the platform base, is cut. The firewall got two openings: HTTPS for the web interface and the REST API, and 3306, the MySQL port, which is what the database user created for API Services below connects to.

$ firewall-cmd --permanent --add-service=https
$ firewall-cmd --permanent --add-port=3306/tcp
$ firewall-cmd --reload

Recording files pile up under /var/log/ocie/recording. A daily cron job deletes those older than 30 days. The procedure is a Config document, Unified Manager 9.4: installation on RHEL 7, and so is the cron job, ocie-recording-cleaning.

Configuration in the web interface

Everything after the RPMs is clicking, and the notes list it screen by screen. The first login as umadmin asks for the mail settings, for AutoSupport and for a new password.

The certificate file is the chain in one piece: the host certificate first, then each intermediate CA, then the root, all PEM. The CSR went to the security team, the same way as for the clusters in Encryption and certificates. Unified Manager is the one component of the storage that could really use the mail server, because it can authenticate; the ONTAP nodes could not, as the previous article shows.

The clusters were added with the local admin account. An older copy of the notes lists a dedicated account ocum on the clusters; in the final notes it is admin, which gives the management server more than it needs.

Users are a mix of directory groups and local accounts.

The account list at the top of the notes and the design both give sensu the role Monitor; the screen-by-screen section says Operator. I cannot tell which was final.

The REST test

Unified Manager 9.4 has a small REST API of its own. The check that the sensu account works was one query for the aggregates.

$ export API_KEY="sensu"
$ export API_PASSWORD="<SENSU_PASSWORD>"
$ curl -u ${API_KEY}:${API_PASSWORD} --basic -X GET --header 'Accept: application/vnd.netapp.object.inventory.hal+json' 'https://dc1-a-vcocm001.adm.example.net/rest/aggregates?limit=20'

{"_embedded":{"netapp:aggregateInventoryList":[
  {"aggregate":{"id":2352,"label":"DC1_A_ANAS001_data1"},"aggregateType":"HDD",
   "availableCapacity":9466.912460327148,"totalCapacity":21937.454948425293,"iops":537.518,"latency":5.65094,
   "utilization":11.783,"status":"OK","cluster":{"label":"DC1-A-XNAS001"},"node":{"label":"DC1-A-ANAS001"}},
  ...
 ]},"totalCount":3}

The answer, abridged here, had three data aggregates, each with capacity, IOPS, latency and a status: what a monitoring check needs.

API Services in a container

Why a container, and why its own machine

The first plan was to run both products in systemd-nspawn containers. The older copy of the Unified Manager notes is exactly that: Unified Manager 7.2P1 on RHEL 7.4, installed into a chroot tree with yum --installroot and booted with systemd-nspawn, and the older design document says so for both virtual machines. The notes do not state the reason in a sentence. What they show is what the products demand of the host: their own MySQL server in place of the baseline's MariaDB libraries, a writable /usr, their own users and init scripts. A container gave the product a plain RHEL userland of its own and left the hardened host as it was.

For Unified Manager that was given up with version 9.4: the final notes install it directly on the host, with the baseline adjusted as described above. API Services stayed in its container, and the header of its notes still gives the address of the Unified Manager machine as its URL, which suggests it was first built there, next to Unified Manager. That could not work the simple way. As a service the container shares the host's network, both products bring a MySQL server, and both want port 3306. The attempt to move one of them is the first dead end below. API Services got its own virtual machine, DC1-A-VCOCM002.

Building the container

The root tree lives on the data disk, here mounted at /chroot. It is built from the host with nothing but RPM and yum: an empty RPM database, the release package, then packages installed into the tree.

$ mkdir -p /chroot/api/var/lib/rpm
$ rpm --root /chroot/api --initdb
$ yumdownloader --destdir=/tmp redhat-release
$ rpm --root /chroot/api -ivh --nodeps /tmp/redhat-release*rpm
$ yum --installroot=/chroot/api -y install zip p7zip unzip systemd dbus mc bind-utils net-tools
$ yum --installroot=/chroot/api -y install java-1.8.0-openjdk
$ yum --installroot=/chroot/api group install "Minimal Install"

The repository definitions for RHEL, EPEL and MySQL Community are copied or installed into the tree, so that yum works inside it later. The container is then started once without booting, to set a root password and to allow logins on pts/0 for machinectl login; SELinux was set to permissive for that step and back to enforcing afterwards.

$ setenforce 0
$ systemd-nspawn -D /chroot/api/ --machine api_container
$ setenforce 1

A second start with -b boots it as a full system, and the SSH daemon inside is moved off port 22. In the notes this test boot has --network-bridge=bridge0 --network-veth; the unit file has no network option, so as a service the container uses the host's network stack and its daemons must not collide with the host's. As a service it is one unit file that runs systemd-nspawn --keep-unit --machine=api_container --directory=/chroot/api -b -j, with Restart=always, started with the host.

$ systemctl add-wants multi-user api_container
$ firewall-cmd --permanent --add-port=8443/tcp
$ firewall-cmd --permanent --add-port=2222/tcp
$ firewall-cmd --reload

The build is a Config document, API Services: the nspawn container, and the unit is another, api_container.service. The notes disagree with themselves about the SSH port of the container: sshd_config gets Port 3333, the firewall and the design open 2222.

Inside the container the internal CA certificates were installed from a prepared bundle, the users umadmin and jboss created, MariaDB-common removed, and the installer run. It asks four questions: whether to install, the password of its admin user, the port of the API (8443, the default) and the port of the JBoss service (8080 in place of the default 80).

The Java keystore

API Services is a Java application and keeps its TLS key in a Java keystore, keystore.jks, with a self-signed certificate. Replacing that with a certificate from the internal CA took more steps than anything else in these notes.

Two later sections are corrections. One changes the password of the private key with keytool -keypasswd; the application reads the key password and the store password from two properties, and presumably what was typed at key generation did not match what had been put into the configuration. The other converts the keystore from Java's own JKS format to PKCS12, in place, after a backup; keytool itself had recommended that in a warning when the key was generated. The procedure, with placeholders, is a Config document: API Services: Java keystore.

The test query

$ export API_KEY="sensu"
$ export API_PASSWORD="<SENSU_PASSWORD>"
$ curl -u ${API_KEY}:${API_PASSWORD} --basic https://dc1-a-vcocm002.adm.example.net:8443/api/4.0/ontap/aggregates --insecure

{"status":{"code":"SUCCESS"},"result":{"total_records":8,"records":[
  {"name":"DC1_A_ANAS001_data1","aggregate_type":"hdd","state":"online","raid_type":"raid_dp",
   "raid_status":"raid_dp, mirrored, normal","mirror_status":"mirrored",
   "size_total":14337925238784,"size_used_percent":79,"is_snaplock":false},
  ...
]}}

Eight aggregates of both clusters, root aggregates included, each with its RAID and mirror state. For a MetroCluster mirror_status is the field a check wants. The --insecure in the recorded query suggests it was run before the signed certificate was in place. The notes do not record the step in the administration interface that connects API Services to Unified Manager; only the two accounts created for it on the Unified Manager side.

Two dead ends

Moving MySQL to another port. Presumably to let both products share a host, MySQL inside the container was moved to port 3360 in my.cnf, and it listened there. Then the application had to be told. The port was changed in the configuration script where OCIE_MYSQL_PORT is set, and the notes list the other files that contain the string 3306. The conclusion is in the notes as written: "I can't define Netapp API services to use MySQL on different port. It seems that using of port 3306 is hardcoded in JAVA code, because I changed lot of config files without success."

User namespaces. The notes of both products end with the kernel argument that enables user namespaces on RHEL 7, user_namespace.enable=1, set with grubby and followed by a reboot, and with the command to take it out again. With user namespaces the container's root would not be the host's root. The link list beside it is about --private-users and setuid problems, the unit file has no such option, and the section sits under "OTHER notes". Whether the container was ever booted that way the notes do not say; as built it ran with the host's user IDs.

What became of it

The older design document (March 2018) describes both products, each with its certificate, its firewall rules and its access table, and leaves two sections for later: "Integration of Sensu with Netapp OnCommand Unified Manager" and "Integration of Sensu with Netapp API services", each with the sentence that it will be completed when the integration is finished. In the final design, version 0.2, API Services is gone: no virtual machine, no certificate, no flows. Only the api-services account in the user table recalls it. The section on integrating Sensu with Unified Manager is still there, and its content is three times the word TODO.

So the Source material shows both REST interfaces answering a query with the sensu account, and no Sensu check that used them. Why the second product disappeared from the design, the design does not say. Unified Manager 9.4 answering REST queries itself, as the test above shows, is one possible reason.

Reading it today

• Unified Manager was renamed. Since version 9.6 it is Active IQ Unified Manager; the current documentation is for 9.18, on RHEL 8.x or 9.6 with MySQL 8.4 and OpenJDK 17, and it recommends 4 CPUs and 12 GB of memory.
• API Services is gone. It reached end of support in April 2021. Its place is taken by the REST APIs of Unified Manager and its API gateway, and by the REST API of ONTAP itself.
• The rule about repositories still stands. The required third-party software must come from the listed repositories only, which is the MariaDB conflict of this article in one sentence.

Lessons

• Vendor software that wants to own the operating system should get one. Half of the work here was the hardened baseline and the product's installer disagreeing about /usr, about MariaDB and about users. A plain RHEL virtual machine for an appliance-like product is cheaper than a container built by hand to hide one from the other.
• A hand-built nspawn container is nobody's supported platform. Red Hat's knowledge base article in the link list asks whether systemd-nspawn is a supported container runtime; the vendor's installer certainly did not expect it.
• Check which port is hardcoded before planning two databases on one host.
• Finish the integration or remove the component. The monitoring user, the firewall rule and the certificate existed for an integration the design never got to describe.