Subject: Re: [OPSAWG] Comments on draft-asai-vmm-mib-05

From: Hirochika Asai <[email-masked]>
Date: Mon, 4 Nov 2013 18:38:11 -0800

Dear Jon,

Thank you for your comments.  I'm sorry for replying late but it took some
time to review the objects you proposed.

First of all, I understand that these objects might be useful, especially for
analyzing the performance problems of virtual machines.  However, I
think we need discussion on these objects because they might be
uncommon to the current implementations of hypervisors.  The objects
in the MIB module should be supported by implementation.

I've checked the current implementation of Xen and KVM whether these
are managed and accessible in these hypervisors.  However, as far as I
know, these objects are not accessible through their API.  IMO, these
objects are not commonly-used-measures at this moment, and
consequently, they are not good to be added to the MIB module now.
We might be able to work on updating the MIB module when they are
implemented and commonly used in future.

If you know any hypervisors providing these measures to operators,
I would like you to share the information and let's discuss whether we'd
add them.

Best regards,
Hirochika

On Oct 24, 2013, at 10:08 AM, Saperia Jon <[email-masked]> wrote:

> I thought two additional objects might be of value based on current experience:
> 
> cpuReadyTime - how much time a VM is waiting for CPU resources.  This could be expressed in milliseconds over the past minute.
> 
> storageLatency - a problem that sometimes comes up is that the CPU is available to the VM but due to storage latency work can not continue.  This object will show average storage latency over a short period of time (perhaps 15 seconds).  
> 
> Notifications should be created for these two objects that can be sent when values exceed configurable limits.
> 
> Thank you 
> /jon
> 
> Jon Saperia
> [email-masked]
> Enterprise Architect
> Harvard University Information Technology
> Innovation & Architecture
> (P) (617) 384-6683
> 1350 Mass Ave., room 758
> Cambridge, MA 02138
> 
> 
> 
> 
> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg

-- 
Hirochika Asai <[email-masked]>, The University of Tokyo

---

From: Michael MacFaden <[email-masked]>
Date: Tue, 5 Nov 2013 10:02:29 -0800 (PST)

Hi,

To add to Jon's request to see 'top' like stats from virtual cpus assigned
to a given virtual machine, I worked recently finished collecting feedback 
for a few US fortune 50 enterprises, here is the compiled the following list of 
metrics they asked for. I think we already have quite a few covered,
but still think some of these should be included in the next draft...

Virtual Machine General Metrics
          MachineName  name of VM
          MachineServer  ESX server that hosts this VM
          MachinePowerStatus  power status of the VM
          MachineUpTime  number of seconds since last boot
          MachineLastUpdate  number of heartbeats received from the VM
          MachineGuestOS  fullname of the guest OS
          MachineGuestIpAddr  IP address of the guest
          MachineCPUNum  Number of vCPUs in SYSTEM
          MachineResourcePool  The Resource Pool this VM is a part of
          MachineLastVMotionTime Shows the last time a VM was vmotion
          MachineHAEligible  Is this Virtual machine protected using Vmware HA
          MachineCreatedBy  Who Created this Virtual Machine
          MachineCreatedDateTime When was this Virtual Machine Created

Virtual Machine Memory
          MemoryTotalSize  memory size of the VM in MB
          MemoryMaxAlloc  amount of memory in MB that the VM can receive
          MemoryMinAlloc  amount of memory in MB guaranteed to the MB
          MemoryHostUsage  consumed host memory in MB
          MemoryGuestUsage  active guest memory in MB
          MemoryHostUtil  host memory used as a percentage of the total VM memory
          MemoryGuestUtil  guest memory used as a percentage of the total VM memory
          MemorySwapToFile  amount of memory being swapped in KB

Virtual Machine Processor
          ProcessorIdleTime  CPU time spend in wait state (milliseconds)
          ProcessorUserTime  CPU time spent on user processes (milliseconds)
          ProcessorSysTime  CPU time spent on system processes (milliseconds)
          ProcessorWaitTime  CPU time spent in ready state (milliseconds)
          ProcessorUtilization  user time as a percentage of total CPU time
          Processor%Ready  ready time as a percentage of total cpu time
  
Virtual Machine Disk
          DiskDescription  disk label
          DiskType  type of disk (hard disk, floppy, etc)
          DiskAccess  read/write
          DiskCapacity  capacity of the virtual disk
          DiskRemovable  true/false

Virtual Machine Partitions
          PartitionLabel  disk path (c:\, /boot, etc)
          PartitionCapacity  capacity of the disk in bytes
          PartitionUsedSpace  capacity minus freespace
          PartitionUtilization  used as a percentage of capacity

Virtual Machine Network
          Description  Network device description
          PhysAddress  physical MAC address
          Read Count  number of packets read
          Write Count  number of packets written
          Read Rate  number of bytes read
          Write Rate  number of bytes written

Virtual Machine Processes
          PID  Process ID
          ProcessName  ProcessName
          ProcessParam  ProcessParameters
          ProcessCPUUsage  ProcessCPUUsage
          ProcessMemoryUsed ProcessMemoryUsage

Next step, break these down from what we have and do not.
Decide what goes where, not all of course has to go into VMM-MIB,
for example reporting the process list of a VM might be best reported
via ENTITY-MIB/HOST-RESOURES-MIB via context.

Regards,
Mike MacFaden
Staff Engineer I R&D Apps, VMware Palo Alto CA

---

From: Saperia Jon <[email-masked]>
Date: Tue, 5 Nov 2013 13:16:11 -0500

I think Mike's additions are very helpful.  I would also like to add a comment about our approach for the development of the MIB Module.  Implementability is a key consideration for any object that we wish to make a standard.  That said, we should not set as a pre-requisite that vendors have already implemented an object, or something like it, before suggesting its addition.  The first thing to consider is: will an object help improve our operational environment?  I suspect that is what drove the responses in the results Mike posted.  I got my addition from talking with folks here at work that run our virtualization environment.

The second point is that the suggestion to reporting certain information via the ENTITY/HOST-RESOURCES ( or I suggest for some items the SYSAPPL module) is a very good one that should be considered.

/jon

Thank you 
/jon

Jon Saperia
[email-masked]
Enterprise Architect

---

From: Hirochika Asai <[email-masked]>
Date: Tue, 5 Nov 2013 10:50:23 -0800

Dear Mike,

Thank you for sharing the feedbacks from operators.  I think some of them are
already defined in the current version, and some of the rest shall be good to be
defined in the next version.  I will review these objects comparing the current
version of our proposal.

At this moment, I don't understand some of them, so could you please explain them?

>          ProcessorSysTime  CPU time spent on system processes (milliseconds)
What's the difference between these two time metrics?
Are they user-time and system-time at a guest OS?

>          ProcessMemoryUsed ProcessMemoryUsage
Do you mean they are processes on a guest OS?

If the objects above are those of a guest OS, they need a special tool such as
vmware tools installed to the guest OS.  So, MIB modules on guest OSes, such
as HOST-RESOURCES-MIB, could be used.

The following objects are also related to guest OSes.
>          PartitionUtilization  used as a percentage of capacity

Some of them might be recognizable from hypervisor.  The partition format
itself does not vary by guest OS (i.e., MBR or GPT partition), so some of these
might be able to available from hypervisor.  However, I still don't think VM-MIB
is the appropriate place to contain these objects, and it is better to use MIB
modules on guest OS (e.g., HOST-RESOURCES-MIB).  Moreover,
PartitionLabel, PartitionUsedSpace, and PartitionUtilization are information
on file system, then a hypervisor cannot recognize them without any special
tools such as vmware tools.

Thank you.
Hirochika

---

From: Michael MacFaden <[email-masked]>
Date: Tue, 5 Nov 2013 23:53:03 -0800 (PST)

> What's the difference between these two time metrics?

Breakdown between apps and OS, can be code running in priviledged processor
mode vs non as well. Helps to know if a driver is going nuts or some web service.

> Are they user-time and system-time at a guest OS?

Yes All of these items presented are about either the Virtual Machine or code running in the 
virtual machine.

> Do you mean they are processes on a guest OS?

Yes, they want that.

> If the objects above are those of a guest OS, they need a special tool such
> as vmware tools installed to the guest OS.  So, MIB modules on guest OSes, such

Yes, that's one way. There could be others. More typically there will be 
SNMP agent running in the Guest OS that is polled separately and/or have an
entry in ENTITY-MIB with hypervisor is another way.
 
> itself does not vary by guest OS (i.e., MBR or GPT partition), so some of
> these  might be able to available from hypervisor.  However, I still don't think
> VM-MIB is the appropriate place to contain these objects, and it is better to use
> MIB modules on guest OS (e.g., HOST-RESOURCES-MIB).  Moreover,

Makes sense to me. In fact I'd prefer if there was prose in the MIB to
explain all how and where we recommend they look for these items.

Regards,
Mike MacFaden

---

From: Jon Saperia <[email-masked]>
Date: Wed, 6 Nov 2013 06:01:15 -0500

Please see one additional comment at the end...

> explain all how and where we recommend they look for these items.

A good proposal.  A number of earlier MIB Documents contained a section call "Relationship to other MIBs".  That would be a good way to address this.

Jon Saperia
Enterprise Architect
(M) (617) 201-2655
(P) (617) 384-6683
1350 Mass Ave., room 758
Cambridge, MA 02138

---

From: Robert Story <[email-masked]>
Date: Wed, 6 Nov 2013 08:09:18 -0500

On Tue, 5 Nov 2013 23:53:03 -0800 (PST) Michael wrote:
MM> [...] Yes All of these items presented are about either the Virtual
MM> Machine or code running in the virtual machine.
MM> 
MM> [...] More typically there will be SNMP agent running in the Guest OS
MM> that is polled separately and/or have an entry in ENTITY-MIB with
MM> hypervisor is another way. 
MM> > 
MM> > [...] However, I still don't think VM-MIB is the appropriate place to
MM> > contain these objects, and it is better to use MIB modules on guest
MM> > OS (e.g., HOST-RESOURCES-MIB).  [...]
MM> 
MM> Makes sense to me. In fact I'd prefer if there was prose in the MIB to
MM> explain all how and where we recommend they look for these items.

I don't think that the MIB is the right place for this. Sounds to me like
what you are looking for is a BCP on monitoring virtual machines.

Robert

--
Senior Software Engineer
Parsons Government Services , National Security & Defense Division

---

From: Michael MacFaden <[email-masked]>
Date: Wed, 6 Nov 2013 07:49:07 -0800 (PST)

>I don't think that the MIB is the right place for this. Sounds to me like

Maybe I'm not making myself clear. 

Take the context of an an engineer trying to implement this MIB module.
Commonly such engineers will not be experts in SNMP or IETF standard data modules.

Take the context of an engineer writing the mgmt app usin said MIB module.
They also might not know where to look for said information.

A best practice would not be useful here as it doesn't provide prescriptive relationship.
And it has been common practice for a given MIB module to explain its relationship to
other MIB modules. This greatly improves interoperability in my experience.

Regards,
Mike MacFaden

---

From: Hirochika Asai <[email-masked]>
Date: Wed, 13 Nov 2013 16:43:12 +0900

Dear Mike and all,

Thank you for your description on several metrics I asked.  The followings
are the correspondence table of metrics you had sent us to the current
definition of VM-MIB.  The metrics that are not contained in VM-MIB is
marked with "*".  However, some of them cannot be obtained at
hypervisors without any support of a special tool such as VMware tools
because they are managed by a guest OS.  These objects are marked
with "#".

Virtual Machine General Metrics
         MachineName  name of VM
             ==> vmName
         MachineServer  ESX server that hosts this VM
             ==> vmHvSoftware etc.
         MachinePowerStatus  power status of the VM
             ==> vmAdminState, vmOperState
         MachineUpTime  number of seconds since last boot
             ==> vmUpTime (But I think we need to clarify that this is uptime since last boot, but not last migration)
         *MachineLastUpdate  number of heartbeats received from the VM
             ==> Need vmware tools? Hypervisors can know whether a CPU is to scheduled to a guest without heartbeat.
         MachineGuestOS  fullname of the guest OS
             ==> vmOSType
         *MachineGuestIpAddr  IP address of the guest
             ==> Need vmware tools? ==> ENTITY-MIB?
         MachineCPUNum  Number of vCPUs in SYSTEM
             ==> The number of physical CPUs: HOST-RESOURCES-MIB at hypervisor
                 The number of virtual CPUs: Count entries in vmCpuTable
         *MachineResourcePool  The Resource Pool this VM is a part of
             ==> What is resource pool? an ID of hypervisor cluster?
         *MachineLastVMotionTime Shows the last time a VM was vmotion
              ==> discussion-general-1: Do we need to have this? e.g., vmLastMigration
         *MachineHAEligible  Is this Virtual machine protected using Vmware HA
              ==> discussion-general-2: I don't think it's to be added to VM-MIB.
         *MachineCreatedBy  Who Created this Virtual Machine
              ==> discussion-general-3: This is needed (as SnmpAdminString)? Or not needed?
         *MachineCreatedDateTime When was this Virtual Machine Created
              ==> discussion-general-4: Whether we add this or not

Virtual Machine Memory
   ====> In this set of metrics, I may misunderstand the difference between host, guest, and VM
         MemoryTotalSize  memory size of the VM in MB
               ==> vmCurMem? What's the difference from MemoryGuestUsage?
         MemoryMaxAlloc  amount of memory in MB that the VM can receive
                ==> vmMaxMem
         MemoryMinAlloc  amount of memory in MB guaranteed to the MB
                 ==> vmMinMem
         *?MemoryHostUsage  consumed host memory in MB
                 ==> What's this?
         MemoryGuestUsage  active guest memory in MB
                 ==> vmCurMem? What's the difference from MemoryTotalSize?
         *MemoryHostUtil  host memory used as a percentage of the total VM memory
                 ==> This can be calculated from vmCurMem and hrMemorySize of HOST-RESOURCES-MIB
         *MemoryGuestUtil  guest memory used as a percentage of the total VM memory
                 ==> This an be calculated from vmCurMem and hrMemorySize of HOST-RESOURCES-MIB?
         *#?MemorySwapToFile  amount of memory being swapped in KB
                 ==> discussion-men-1: Hypervisor cannot know the swap size on the guest OS without a special tool, i.e., this metrics is to be obtained from the agent on guest OSes. So does VMM-MIB need not to have this?
                     If this means the memory managed by a hypervisor is swapped to the hypervisor's storage, we need another discussion.

Virtual Machine Processor
         *ProcessorIdleTime  CPU time spend in wait state (milliseconds)
                  ==> discussion-cpu-1: (wait state?) Related to Jon's proposal
         #ProcessorUserTime  CPU time spent on user processes (milliseconds)
         #ProcessorSysTime  CPU time spent on system processes (milliseconds)
                  ==> vmCpuTime (user&sys are combined, because hypervisors do not distinguish these on a guest OS.)
         *ProcessorWaitTime  CPU time spent in ready state (milliseconds)
                  ==> discussion-cpu-1: (ready state?) Related to Jon's proposal
         *ProcessorUtilization  user time as a percentage of total CPU time
                  ==> (calculated from vmCpuTime)
         *Processor%Ready  ready time as a percentage of total cpu time
                  ==> (calculated from ProcessorWaitTime if newly defined)

Virtual Machine Disk
         DiskDescription  disk label
                   ==> vmStorageResourceID?
         DiskType  type of disk (hard disk, floppy, etc)
                   ==> vmStorageMediaType (but floopy is not defined), vmStorageMediaTypeString
         DiskAccess  read/write
                   ==> vmStorageAccess
         DiskCapacity  capacity of the virtual disk
                   ==> vmStorageDefinedSize, vmStorageAllocatedSize
         *DiskRemovable  true/false
                   ==> discussion-storage-1: whether we need this or not

Virtual Machine Partitions
         *#PartitionLabel  disk path (c:\, /boot, etc)
         *#PartitionCapacity  capacity of the disk in bytes
         *#PartitionUsedSpace  capacity minus freespace
         *#PartitionUtilization  used as a percentage of capacity
               ==> (should be HOST-RESOURCES-MIB on guest OS?)

Virtual Machine Network
         Description  Network device description
              ==> ifDescr or ifName from vmNetworIfIndex
         PhysAddress  physical MAC address
              ==> vmNetworkPhysAddress
         Read Count  number of packets read
              ==> ifOutUcastPkts (ifInUcastPkts?) from vmNetworIfIndex
         Write Count  number of packets written
              ==> ifInUcastPkts (ifOutUcastPkts?) from vmNetworkIfIndex
         Read Rate  number of bytes read
              ==> ifOutOctets (ifInOctets?) from vmNetworIfIndex
         Write Rate  number of bytes written
              ==> ifInOctets (ifOutOctets?) from vmNetworIfIndex

Virtual Machine Processes
         *#PID  Process ID
         *#ProcessName  ProcessName
         *#ProcessParam  ProcessParameters
         *#ProcessCPUUsage  ProcessCPUUsage
         *#ProcessMemoryUsed ProcessMemoryUsage
               ==> (should be HOST-RESOURCES-MIB on guest OS?)

Thank you,
Hirochika

---

From: Saperia Jon <[email-masked]>
Date: Wed, 13 Nov 2013 07:58:38 -0500

I think Mike has it right as I noted earlier.  A section of this module that describes the relationship of this to other relevant MIB modules would be helpful, not only to the two groups (MIB and Management Application Engineers) but also to people that run networks since they are also not commonly standards experts.
/jon

Thank you 
/jon

Jon Saperia
[email-masked]
Enterprise Architect