Ensuring Reliable Workbench/Agent Communication
The working subtitle for this was “Workbenches from Mars, Agents from Venus”. It’s a discussion of what to do if agents and workbenches are not communicating as expected.
The execution of a DevOps Test Performance (formerly Rational Performance Tester) schedule frequently relies on the use of remote agents to distribute workload across multiple resources. Naturally, there is a need for these machines to connect on the testing team’s network. That is not possible unless we ensure that agents and workbenches are communicating.
Bird’s Eye View of How Agents Work
An agent makes use of a Service/Process called MajordomoService, commonly truncated as "Majordomo". Majordomo’s purpose is to manage agent availability. At a very high level, Majordomo broadcasts agent details to one or more workbenches. Those workbenches can then make use of the agent services in executing schedules. Among the details that agents provide to workbenches:
- IP address
- Host name
- Agent software version
- Agent licensing type
Many of these details (but not all) come from the majordomo.config file.
<MajordomoConfig xmlns="http://www.example.org/MajordomoConfiguration">
<debug>false</debug>
<canonicalHostName></canonicalHostName>
<ipAddress></ipAddress>
<pollInterval>10</pollInterval>
<workbenches>
<hostName>localhost</hostName>
<port>7080</port>
</workbenches>
<servers>
<hostName></hostName>
<port></port>
<token></token>
<slug></slug>
</servers>
</MajordomoConfig>
When the agent polls the workbench, it sends a request similar to:
GET /agent/whatisthybidding?canonicalHostName=AGENT01.HCL.COM
&ip=10.111.11.11&poll=10000&version=10200&engineListSize=0
&agentLicenseType=Windows+8&osName=Windows+10&arch=amd64&ram=8350035968 HTTP/1.1
to the workbench. The canonicalHostName in this request is determined by Java code like:
InetAddress.getLocalHost().getCanonicalHostName()
unless a canonicalHostName is specified in the majordomo.config.
Likewise, the agent determines the IP value in the request using Java code like:
InetAddress.getLocalHost().getHostAddress()
unless the majordomo.config specifies an ipAddress.
If the canonicalHostName and ipAddress fields in majordomo.config are not empty, the agent will send whatever strings are specified, even if they are not actually names or IP addresses. The Agent Status dialog on the workbench will display whatever strings are supplied in the GET request.
How Workbenches Resolve Network Details to Work with Agents
When a schedule specifies playing back on a location, the workbench will get the IP address corresponding to that location by resolving the "Host Name or IP Address" specified in the Location details. The workbench will then compare that IP address to the list of IP's supplied by the agents which have contacted it. If a match is found, that agent will be used to run the test. If the Host Name is a valid IP address ("valid" means that it is formatted correctly; it doesn't have to be the correct IP of the location), the workbench will use that IP address to compare against the list of IP's supplied by the agents.
If the location's Host Name (MyAgentName for example) resolves but there is no agent connecting from that IP address, the workbench will report:
Agent MyAgentName not ready, time of last contact: No successful contact
Check Agents Failed
Alternatively, if the Host Name of the location cannot be resolved, the error message is:
Unknown host 'MyAgentName'
Check Agents Failed
Note that it is possible to play back on an agent that is supplying an incorrect IP address provided that the Host Name of the location is resolved to that address on the workbench.
For purposes of matching an agent to a location, the canonicalHostName is not relevant for the workbench.
Unusual Circumstances
Typically, testing environments use consistent Internet Protocol versions, either IPv4 or IPv6. That means the IP address broadcast by the agent appears in the same protocol version as that which the workbench resolved from the "Host Name or IP Address" specified in the Location details.
We recently encountered a scenario where this consistency was lacking. Agents were broadcasting IPv6 information. On the workbenches, the Locations’ Host Names were resolving to IPv4 formats. This led to:
Agent MyAgentName not ready, time of last contact: No successful contact
Check Agents Failed
Even though the agent was there, waiting for work. The workbench couldn’t find it because the two machines were speaking a different language, or rather, using a different Internet Protocol version format.
Proper Solution
There are networking-level solutions to this. Life is a lot less confusing if all the machines in your testing environments are using the same Internet Protocol version.
Of course, that may not be something that testers can influence with any sort of immediacy.
The next best thing is to force the agent to broadcast in a format the workbench can resolve. Update the ipAddress value in the majordomo.config file to use the proper ipAddress value and syntax (IPv4 vs IPv6).
<MajordomoConfig xmlns="http://www.example.org/MajordomoConfiguration">
<debug>false</debug>
<canonicalHostName></canonicalHostName>
<ipAddress>10.0.2.0</ipAddress>
…
</MajordomoConfig>
The 10.0.2.0 value will appear in the workbenches’ Agent Status dialog AND the internal table of IPs supplied by the agents.
Alternative Solution
If you cannot update the majordomo.config file, there is another, more workbench-oriented solution.
Let us suppose that the agent is broadcasting IPv6 values while the workbench uses IPv4 values. The request to the workbench from the agent looks like this:
GET /agent/whatisthybidding?canonicalHostName=AGENT01.HCL.COM
&ip=2610:30:4162:118:358b:1bf5:1217:4a75&poll=10000&version=10200&engineListSize=0&agentLicenseType=Windows+8&osName=Windows+10&arch=amd64&ram=8350035968 HTTP/1.1
The workbench’s internal table entry will reflect an IPv6 address. However, because the workbench resolves to IPv4 addresses, the location detail will never match the internal table.
To compensate for this, you can create an entry in the hosts file (C:\Windows\System32\drivers\etc\hosts) like:
2610:30:4162:118:358b:1bf5:1217:4a75 location01
The workbench will try to resolve “location01”. The presence of this line in the hosts file will force that value to 2610:30:4162:118:358b:1bf5:1217:4a75. The IPv6 value matches a value in the internal table of agents. Work will be directed to that agent.
What Did We Learn?
Ideally, all the devices related to your testing environment are using the same default IP address format. In the off chance that they are not, it’s not the end of the world. Editing the majordomo.config file on your agents allows you to explicitly specify the desired IP format. If that is not an option, you can edit the etc/hosts file on the workbench.
Taken to the Extreme
If you read between the lines, you may have noticed something: with the right set of changes, it’s possible to identify your agents with ANY name you want.
Let’s take a typical scenario where we have an agent installed on the same machine as our workbench. The initial majordomo.config would look like this:
<MajordomoConfig xmlns="http://www.example.org/MajordomoConfiguration">
<debug>false</debug>
<canonicalHostName></canonicalHostName>
<ipAddress></ipAddress>
<pollInterval>10</pollInterval>
<workbenches>
<hostName>localhost</hostName>
<port>7080</port>
</workbenches>
<servers>
<hostName></hostName>
<port></port>
<token></token>
<slug></slug>
</servers>
</MajordomoConfig>
On the workbench, if we look at the AgentStatus, we’d see this:
Sorry, have to redact the actual hostnames. Security, ya know?
As you can see, the IP address is presented as 9.25.0.109.
We can change that in the majordomo.config file. Let’s go with this:
<canonicalHostName>NOCLIST</canonicalHostName>
<ipAddress>EthanHunt</ipAddress>
Now the Agent Status details for the same agent look like this:
That’s pretty cool. “EthanHunt” will be saved in the internal table used by workbench. Now, how do we convince the workbench that “EthanHunt” is a real agent?
I have a VU Schedule with a User Group that I want to run on my agent. I create a Location for my agent.
We just need to make NOCLIST resolve to EthanHunt. This is done in the hosts file.
The super spy analogy doesn’t hold up, if you really think about it, but it does represent the flexibility available for unusual circumstances.
#RationalPerformanceTester