Quality data is paramount in any TBM implementation. Good data might make the difference between a convincing or a sloppy presentation, leaving stakeholders with a good impression, or a feeling of disbelief. In this episode, Bill goes through the data sources in their TBM shop, describing the various types and automation possibilities.
***
"Hey, Bill. I'd need to go through our data sources for the IT environment." Said Ellie Nakamura, the organisation's finance TBM Analyst (TBMA).
"How's that, Ellie? You pretty much know everything we have…"
"Well, our friends in Controlling are working with some external consultants, and they are making an inventory. They are also asking about automation, whether already available, or whatever we could do in the future. I have already compiled the financial sources, but I'd need a hand with the technical stuff."
"Sure, Ellie. Where do we start? OK, first things first: the Configuration Management Data Base."
"Good ol' CMDB, yes, what do we get from it?"
"We have two data sets: one has all the hardware devices, including servers, storage, network, appliances and most of the virtual machines we run."
"How is the data updated, Bill?"
"On the one hand, the CMDB team has a discovery mechanism, an off-the-shelf product that scans the environment and records the data the devices report. Not sure how it works, but that's as much as I need to know."
"And how do we get it into the system?"
"Every day, the CMDB team runs a script that extracts the data and creates a file. We pick up the latest version of that file when we're ready to close the month. The data set has many columns, not only with hardware details but also with metadata such as owner team, status, installation date, and many more."
"Do you really need all of that information?"
"We just need a few columns. Off the top of my head, the Configuration Item (CI) name, the owner group, for servers, the number of cores, be them physical or virtual, and the status. The latter allows us to distinguish between active devices, and those that are on the way in or out of our data center. We get rid of those."
"You mentioned another data source from the CMDB, Bill. What is that?"
"Probably as important as the asset register, we load another extract that has links between servers and application components. The link between servers and applications is critical for us. When we started we didn't have this, so we had to go one by one, trying to figure out what the machine related to, by looking at other attributes, including free-form fields. The original server-to-application mapping table used to have over 15,000 entries. Not funny, let me tell you."
"So now it's working better…" Said Ellie.
"Much better. I haven't checked the numbers recently, but we might be in a Pareto-like situation, with roughly 80% of the servers having links to applications. With a bit of creativity, we can overcome the gap."
"What do you mean, Bill?"
"Well, it reminds me of something our TBM engine's pre-sales consultant told us during the Request For Proposal (RFP). Gina Marsden told us we wouldn't believe what we could do with the data we had-and she was right!"
"So how do you bridge that gap? 20% sounds like a big hole…"
We have some ancillary data sources provided by some application management groups. They are working towards integrating the information in the CMDB, but until then, we use another set of extracts from their monitoring systems."
"Isn't that a lot of overhead?"
"I guess. The good news is that we only have to load once a month, so it's doable. Also, having or missing an extra server, or application link, it will not make a big difference." Said Bill, smiling.
"Could you automate any of that, Bill?"
"In principle, yes. The limiting factor is the fact that our TBM engine is a Software-as-a-Service (SaaS) application. As such, it is hosted outside our premises. This makes data exchanges problematic."
"But how does it work when you upload the data manually?"
"Since we do it since our workstations or laptops, it is not a problem, as they have access to the internet as long as we use standard ports. The data exchange product the vendor has is powerful, but we need to set it up properly. It's something we are working on, but with point solutions for now."
"Point solutions?"
"We helped our internal IT department automate the load of their time reports, by using Secure FTP (SFTP). It took us several months until we figured out all the technical details. They put an SFTP client program on the server running the time reports, and we had to open the required address ranges and ports in our firewalls. Eventually, the first file transfer worked, dropping the report on a sandbox. The provider's software would then read the file and load to the appropriate table for the corresponding period in the system."
"Don't you have access to that server?"
"Not anymore. It really belonged to them. We need to get one, but right now we don't have any available. There are other options, like using our workstations via a locally deployed agent software. Again, we're struggling with the technical setup. I just need some time and spend a few days ironing out the details. What takes longest is getting any firewall configuration changes through. Security scrutinises these very closely, and there are specific execution windows for such actions. It'll work-eventually."
"So, everything is manual except that SFTP stuff?"
"The one thing that we did was automating the downloads of the public Cloud service providers (CSPs). We get billing information from them, and it works like a charm. Also, by now the billing data has millions of lines, so it would entail downloads of several hundred megabytes, or even beyond 1 GB."
"Wasn't that complicated, Bill?"
It depends. With public Cloud, the tricky bit is getting the proper access credentials. We had to go back and forth a few times with our vendor and the respective CSP technicians, but we got it to work. We've been doing this for years now."
"What else is there, beyond time reports, Cloud and CMDB data?"
"Very closely related to the server data is an extract of our virtualization platform, that links virtual machines (VMs) and the physical host they run on. VMs inherit the costs of the host, so we need that link."
"And how do you allocate the hosts' costs to the guests?"
"By the number of virtual CPUs, or vCores." Every guest receives an amount proportional to its potential consumption. It works well."
"Thanks, Bill. That's a lot already. What else is there?"
"We update the list of applications and tools every now and then, but not a monthly cycle. Whenever we realize we're missing something. The application catalog is not that reliable, and we have to complement it with additional entries."
"And the business services?" Asked Ellie.
"That's pretty much on demand. The service people have a kind of introduction process, and as part of that we get a work order to add entries as required."
"Coming back to the Cloud topic, what else is there apart from the billing information?"
Bill said: "we need to update the tenant mapping tables for each CSP. Again, this is an ad hoc update, as not always do we find new tenant accounts. I'd prefer a process-driven approach, like for the business services, but I guess it would be a considerable overhead for the Cloud Center of Excellence (CCoE)."
"That's public. What about private Cloud, Bill?"
"We need to update the quota and consumption files for the various fabrics. Similarly, we load the Cloud Management Middleware Software (CMMS) PaaS instance quotas and consumption files. It's several small comma-separated files (CSVs). We'll automate these as soon as we sort out the logistics for the automation."
"Is that it?"
"We're getting closer, Ellie. We still have the storage consumption and the server utilization data. These are recent additions to our portfolio of data sources. They provide very useful insights for on-prem application optimization. Very manual process, but we might get a REST API to automate it. Still in the cards for now."
"Last but not least, transaction volumes. We get two sets of data: one links applications and business services, allowing us to allocate costs in a consumption-based model. The catch? We only get four hours' worth of data in peak time. It's good for the 1:N relationships between Apps and business services, but not much more than that."
"So, what's the other transaction load for?"
"That's another recent addition. They built some kind of script that polls the logging servers and brings back the number of transactions by application. We're still learning about this, but it's becoming very popular: many managers want to come up with KPIs based on transaction unit costs, etc. Another load that we may or may not automate."
"Wow, Bill! That was great."
"I hope it helps, Ellie."
***
#bill-the-tbm-guy