• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

virtuallyGhetto

  • About
  • Privacy
  • VMware Cloud
  • Home Lab
  • Nested Virtualization
  • Automation
    • VMware Kickstart
    • VMware API/SDK/CLI
    • VMware vMA/VIMA
    • VMware OVF / OVFTOOL
  • Apple Mac
  • VCSA
  • VSAN

vob

New VOBs for creating vCenter Server alarms in vSphere 6.0

03/02/2015 by William Lam Leave a Comment

Here are some new VOBs in vSphere 6.0 that I recently came across which can be useful on getting notified on specific events such failed login attempts in ESXi or detecting a device has gone offline in VSAN as some examples. These VOBs can be used to create vCenter Server alarms to take various actions such as a simple UI notification in the vSphere Web/C# Client to sending an email or SNMP trap regarding the event. For more information on how create vCenter Server alarms using VOBs, please take a look at these two articles here and here which also includes a comprehensive list of past vSphere VOBs in vSphere 5.5 which are still applicable in vSphere 6.0.

General vSphere 6.0 VOBs

VOB ID VOB Description
esx.audit.account.locked Remote access for an ESXi local user account has been locked temporarilly due to multiple failed login attempts.
esx.audit.account.loginfailures Multiple remote login failures detected for an ESXi local user account.
esx.audit.esxcli.host.restart Rebooting host through esxcli
esx.audit.lockdownmode.exceptions.changed List of lockdown exception users has been changed.
esx.problem.coredump.copyspace The free space available in default coredump copy location is insufficient to copy new coredumps.
esx.problem.coredump.extraction.failed.nospace The given partition has insufficient amount of free space to extract the coredump.
esx.problem.dhclient.lease.offered.error No expiry time on offered DHCP lease.
esx.problem.pageretire.selectedbutnotretired.high Number of host physical memory pages that have been selected for retirement but could not yet be retired is high.
esx.problem.swap.systemSwap.isPDL.cannot.remove System swap at path {1} was affected by the PDL of its datastore and was removed. System swap has been reconfigured.
esx.problem.swap.systemSwap.isPDL.removed.reconfig.failure System swap at path {1} was affected by the PDL of its datastore. It was removed but the subsequent reconfiguration failed.
esx.problem.vmfs.ats.incompatibility.detected Multi-extent ATS-only VMFS Volume unable to use ATS
esx.problem.vmfs.lockmode.inconsistency.detected Inconsistent VMFS lockmode detected.
esx.problem.vmfs.spanned.lockmode.inconsistency.detected Inconsistent VMFS lockmode detected on spanned volume.
esx.problem.vmfs.spanstate.incompatibility.detected Incompatible VMFS span state detected.
esx.vFlash.VFlashResourceCapacityExtendedEvent vFlash resource capacity is extended
vprob.vmfs.heartbeat.corruptondisk VMFS Heartbeat Corruption Detected

VSAN 6.0 VOBs

VOB ID VOB Description
esx.audit.vsan.net.vnic.added Virtual SAN virtual NIC has been added.
esx.audit.vsan.net.vnic.deleted Virtual SAN network configuration has been removed.
esx.problem.vob.vsan.dom.lsefixed Virtual SAN detected and fixed a medium error on disk.
esx.problem.vob.vsan.dom.nospaceduringresync Resync encountered no space error
esx.problem.vob.vsan.lsom.disklimit2 Failed to add disk to disk group.
esx.problem.vsan.dom.init.failed.status Virtual SAN Distributed Object Manager failed to initialize
vprob.vob.vsan.pdl.offline Virtual SAN device has gone offline.
Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: ESXi, VSAN, vSphere 6.0 Tagged With: alarm, vob, VSAN, vSphere 6.0

Other handy vSphere VOBs for creating vCenter Alarms

04/24/2014 by William Lam 8 Comments

Lately I have been writing on a variety of topics regarding the use of VOBs (VMkernel Observations) for creating useful vCenter Alarms such as:

  • Handy VSAN VOBs for creating vCenter Alarms
  • How to create vCenter Alarm to alert on ESXi 5.5u1 NFS APD issue?
  • How to automatically monitor VSAN Component threshold using a vCenter Alarm?
  • Detecting A Duplicate IP Address For Your ESXi Hosts Using a vCenter Alarm
  • Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm

I figure it would also be useful to collect a list of all the vSphere VOBs, at least from what I can gather by looking at /usr/lib/vmware/hostd/extensions/hostdiag/locale/en/event.vmsg on the latest version of ESXi. The list below is quite extensive, there are a total of 308 vSphere VOBs not including the VSAN VOBs in my previous articles. For those those of you who use vSphere Replication, you may also find a couple of handy ones in the list.

VOB ID VOB Description
ad.event.ImportCertEvent Import certificate success
ad.event.ImportCertFailedEvent Import certificate failure
ad.event.JoinDomainEvent Join domain success
ad.event.JoinDomainFailedEvent Join domain failure
ad.event.LeaveDomainEvent Leave domain success
ad.event.LeaveDomainFailedEvent Leave domain failure
com.vmware.vc.HA.CreateConfigVvolFailedEvent vSphere HA failed to create a configuration vVol for this datastore and so will not be able to protect virtual machines on the datastore until the problem is resolved. Error: {fault}
com.vmware.vc.HA.CreateConfigVvolSucceededEvent vSphere HA successfully created a configuration vVol after the previous failure
com.vmware.vc.HA.DasHostCompleteDatastoreFailureEvent Host complete datastore failure
com.vmware.vc.HA.DasHostCompleteNetworkFailureEvent Host complete network failure
com.vmware.vc.VmCloneFailedInvalidDestinationEvent Cannot complete virtual machine clone.
com.vmware.vc.VmCloneToResourcePoolFailedEvent Cannot complete virtual machine clone.
com.vmware.vc.VmDiskConsolidatedEvent Virtual machine disks consolidation succeeded.
com.vmware.vc.VmDiskConsolidationNeeded Virtual machine disks consolidation needed.
com.vmware.vc.VmDiskConsolidationNoLongerNeeded Virtual machine disks consolidation no longer needed.
com.vmware.vc.VmDiskFailedToConsolidateEvent Virtual machine disks consolidation failed.
com.vmware.vc.datastore.UpdateVmFilesFailedEvent Failed to update VM files
com.vmware.vc.datastore.UpdatedVmFilesEvent Updated VM files
com.vmware.vc.datastore.UpdatingVmFilesEvent Updating VM Files
com.vmware.vc.ft.VmAffectedByDasDisabledEvent Fault Tolerance VM restart disabled
com.vmware.vc.guestOperations.GuestOperation Guest operation
com.vmware.vc.guestOperations.GuestOperationAuthFailure Guest operation authentication failure
com.vmware.vc.host.clear.vFlashResource.inaccessible Host's virtual flash resource is accessible.
com.vmware.vc.host.clear.vFlashResource.reachthreshold Host's virtual flash resource usage dropped below the threshold.
com.vmware.vc.host.problem.vFlashResource.inaccessible Host's virtual flash resource is inaccessible.
com.vmware.vc.host.problem.vFlashResource.reachthreshold Host's virtual flash resource usage exceeds the threshold.
com.vmware.vc.host.vFlash.VFlashResourceCapacityExtendedEvent Virtual flash resource capacity is extended
com.vmware.vc.host.vFlash.VFlashResourceConfiguredEvent Virtual flash resource is configured on the host
com.vmware.vc.host.vFlash.VFlashResourceRemovedEvent Virtual flash resource is removed from the host
com.vmware.vc.host.vFlash.defaultModuleChangedEvent Default virtual flash module is changed to {vFlashModule} on the host
com.vmware.vc.host.vFlash.modulesLoadedEvent Virtual flash modules are loaded or reloaded on the host
com.vmware.vc.npt.VmAdapterEnteredPassthroughEvent Virtual NIC entered passthrough mode
com.vmware.vc.npt.VmAdapterExitedPassthroughEvent Virtual NIC exited passthrough mode
com.vmware.vc.vcp.FtDisabledVmTreatAsNonFtEvent FT Disabled VM protected as non-FT VM
com.vmware.vc.vcp.FtFailoverEvent Failover FT VM due to component failure
com.vmware.vc.vcp.FtFailoverFailedEvent FT VM failover failed
com.vmware.vc.vcp.FtSecondaryRestartEvent Restarting FT secondary due to component failure
com.vmware.vc.vcp.FtSecondaryRestartFailedEvent FT secondary VM restart failed
com.vmware.vc.vcp.NeedSecondaryFtVmTreatAsNonFtEvent Need secondary VM protected as non-FT VM
com.vmware.vc.vcp.TestEndEvent VM Component Protection test ends
com.vmware.vc.vcp.TestStartEvent VM Component Protection test starts
com.vmware.vc.vcp.VcpNoActionEvent No action on VM
com.vmware.vc.vcp.VmDatastoreFailedEvent Virtual machine lost datastore access
com.vmware.vc.vcp.VmNetworkFailedEvent Virtual machine lost VM network accessibility
com.vmware.vc.vcp.VmPowerOffHangEvent VM power off hang
com.vmware.vc.vcp.VmRestartEvent Restarting VM due to component failure
com.vmware.vc.vcp.VmRestartFailedEvent Virtual machine affected by component failure failed to restart
com.vmware.vc.vcp.VmWaitForCandidateHostEvent No candidate host to restart
com.vmware.vc.vm.VmStateFailedToRevertToSnapshot Failed to revert the virtual machine state to a snapshot
com.vmware.vc.vm.VmStateRevertedToSnapshot The virtual machine state has been reverted to a snapshot
com.vmware.vc.vmam.AppMonitoringNotSupported Application Monitoring Is Not Supported
com.vmware.vc.vmam.VmAppHealthMonitoringStateChangedEvent vSphere HA detected application heartbeat status change
com.vmware.vc.vmam.VmAppHealthStateChangedEvent vSphere HA detected application state change
com.vmware.vc.vmam.VmDasAppHeartbeatFailedEvent vSphere HA detected application heartbeat failure
esx.audit.agent.hostd.started VMware Host Agent started
esx.audit.agent.hostd.stopped VMware Host Agent stopped
esx.audit.dcui.defaults.factoryrestore Restoring factory defaults through DCUI.
esx.audit.dcui.disabled The DCUI has been disabled.
esx.audit.dcui.enabled The DCUI has been enabled.
esx.audit.dcui.host.reboot Rebooting host through DCUI.
esx.audit.dcui.host.shutdown Shutting down host through DCUI.
esx.audit.dcui.hostagents.restart Restarting host agents through DCUI.
esx.audit.dcui.login.failed Login authentication on DCUI failed
esx.audit.dcui.login.passwd.changed DCUI login password changed.
esx.audit.dcui.network.factoryrestore Factory network settings restored through DCUI.
esx.audit.dcui.network.restart Restarting network through DCUI.
esx.audit.esxcli.host.poweroff Powering off host through esxcli
esx.audit.esxcli.host.reboot Rebooting host through esxcli
esx.audit.esximage.hostacceptance.changed Host acceptance level changed
esx.audit.esximage.install.novalidation Attempting to install an image profile with validation disabled.
esx.audit.esximage.install.securityalert SECURITY ALERT: Installing image profile.
esx.audit.esximage.profile.install.successful Successfully installed image profile.
esx.audit.esximage.profile.update.successful Successfully updated host to new image profile.
esx.audit.esximage.vib.install.successful Successfully installed VIBs.
esx.audit.esximage.vib.remove.successful Successfully removed VIBs
esx.audit.host.boot Host has booted.
esx.audit.host.maxRegisteredVMsExceeded The number of virtual machines registered on the host exceeded limit.
esx.audit.host.stop.reboot Host is rebooting.
esx.audit.host.stop.shutdown Host is shutting down.
esx.audit.lockdownmode.disabled Administrator access to the host has been enabled.
esx.audit.lockdownmode.enabled Administrator access to the host has been disabled.
esx.audit.maintenancemode.canceled The host has canceled entering maintenance mode.
esx.audit.maintenancemode.entered The host has entered maintenance mode.
esx.audit.maintenancemode.entering The host has begun entering maintenance mode.
esx.audit.maintenancemode.exited The host has exited maintenance mode.
esx.audit.net.firewall.config.changed Firewall configuration has changed.
esx.audit.net.firewall.disabled Firewall has been disabled.
esx.audit.net.firewall.enabled Firewall has been enabled for port.
esx.audit.net.firewall.port.hooked Port is now protected by Firewall.
esx.audit.net.firewall.port.removed Port is no longer protected with Firewall.
esx.audit.net.lacp.disable LACP disabled
esx.audit.net.lacp.enable LACP eabled
esx.audit.net.lacp.uplink.connected uplink is connected
esx.audit.shell.disabled The ESXi command line shell has been disabled.
esx.audit.shell.enabled The ESXi command line shell has been enabled.
esx.audit.ssh.disabled SSH access has been disabled.
esx.audit.ssh.enabled SSH access has been enabled.
esx.audit.usb.config.changed USB configuration has changed.
esx.audit.uw.secpolicy.alldomains.level.changed Enforcement level changed for all security domains.
esx.audit.uw.secpolicy.domain.level.changed Enforcement level changed for security domain.
esx.audit.vmfs.lvm.device.discovered LVM device discovered.
esx.audit.vmfs.volume.mounted File system mounted.
esx.audit.vmfs.volume.umounted LVM volume un-mounted.
esx.clear.coredump.configured A vmkcore disk partition is available and/or a network coredump server has been configured.  Host core dumps will be saved.
esx.clear.coredump.configured2 At least one coredump target has been configured. Host core dumps will be saved.
esx.clear.net.connectivity.restored Restored network connectivity to portgroups
esx.clear.net.dvport.connectivity.restored Restored Network Connectivity to DVPorts
esx.clear.net.dvport.redundancy.restored Restored Network Redundancy to DVPorts
esx.clear.net.lacp.lag.transition.up lag transition up
esx.clear.net.lacp.uplink.transition.up uplink transition up
esx.clear.net.lacp.uplink.unblocked uplink is unblocked
esx.clear.net.redundancy.restored Restored uplink redundancy to portgroups
esx.clear.net.vmnic.linkstate.up Link state up
esx.clear.scsi.device.io.latency.improved Scsi Device I/O Latency has improved
esx.clear.scsi.device.state.on Device has been turned on administratively.
esx.clear.scsi.device.state.permanentloss.deviceonline Device that was permanently inaccessible is now online.
esx.clear.storage.apd.exit Exited the All Paths Down state
esx.clear.storage.connectivity.restored Restored connectivity to storage device
esx.clear.storage.redundancy.restored Restored path redundancy to storage device
esx.problem.3rdParty.error A 3rd party component on ESXi has reported an error.
esx.problem.3rdParty.information A 3rd party component on ESXi has reported an informational event.
esx.problem.3rdParty.warning A 3rd party component on ESXi has reported a warning.
esx.problem.apei.bert.memory.error.corrected A corrected memory error occurred
esx.problem.apei.bert.memory.error.fatal A fatal memory error occurred
esx.problem.apei.bert.memory.error.recoverable A recoverable memory error occurred
esx.problem.apei.bert.pcie.error.corrected A corrected PCIe error occurred
esx.problem.apei.bert.pcie.error.fatal A fatal PCIe error occurred
esx.problem.apei.bert.pcie.error.recoverable A recoverable PCIe error occurred
esx.problem.application.core.dumped An application running on ESXi host has crashed and a core file was created.
esx.problem.boot.filesystem.down Lost connectivity to the device backing the boot filesystem
esx.problem.coredump.capacity.insufficient The storage capacity of the coredump targets is insufficient to capture a complete coredump.
esx.problem.coredump.unconfigured No vmkcore disk partition is available and no network coredump server has been configured.  Host core dumps cannot be saved.
esx.problem.coredump.unconfigured2 No coredump target has been configured. Host core dumps cannot be saved.
esx.problem.cpu.amd.mce.dram.disabled DRAM ECC not enabled. Please enable it in BIOS.
esx.problem.cpu.intel.ioapic.listing.error Not all IO-APICs are listed in the DMAR. Not enabling interrupt remapping on this platform.
esx.problem.cpu.mce.invalid MCE monitoring will be disabled as an unsupported CPU was detected. Please consult the ESX HCL for information on supported hardware.
esx.problem.cpu.smp.ht.invalid Disabling HyperThreading due to invalid configuration: Number of threads: {1} Number of PCPUs: {2}.
esx.problem.cpu.smp.ht.numpcpus.max Found {1} PCPUs but only using {2} of them due to specified limit.
esx.problem.cpu.smp.ht.partner.missing Disabling HyperThreading due to invalid configuration: HT partner {1} is missing from PCPU {2}.
esx.problem.dhclient.lease.none Unable to obtain a DHCP lease.
esx.problem.dhclient.lease.offered.noexpiry No expiry time on offered DHCP lease.
esx.problem.esximage.install.error Could not install image profile.
esx.problem.esximage.install.invalidhardware Host doesn't meet image profile hardware requirements.
esx.problem.esximage.install.stage.error Could not stage image profile.
esx.problem.hardware.acpi.interrupt.routing.device.invalid Skipping interrupt routing entry with bad device number: {1}. This is a BIOS bug.
esx.problem.hardware.acpi.interrupt.routing.pin.invalid Skipping interrupt routing entry with bad device pin: {1}. This is a BIOS bug.
esx.problem.hardware.ioapic.missing IOAPIC Num {1} is missing. Please check BIOS settings to enable this IOAPIC.
esx.problem.host.coredump An unread host kernel core dump has been found.
esx.problem.hostd.core.dumped Hostd crashed and a core file was created.
esx.problem.iorm.badversion Storage I/O Control version mismatch
esx.problem.iorm.nonviworkload Unmanaged workload detected on SIOC-enabled datastore
esx.problem.migrate.vmotion.default.heap.create.failed Failed to create default migration heap
esx.problem.migrate.vmotion.server.pending.cnx.listen.socket.shutdown Error with migration listen socket
esx.problem.net.connectivity.lost Lost Network Connectivity
esx.problem.net.dvport.connectivity.lost Lost Network Connectivity to DVPorts
esx.problem.net.dvport.redundancy.degraded Network Redundancy Degraded on DVPorts
esx.problem.net.dvport.redundancy.lost Lost Network Redundancy on DVPorts
esx.problem.net.e1000.tso6.notsupported No IPv6 TSO support
esx.problem.net.fence.port.badfenceid Invalid fenceId configuration on dvPort
esx.problem.net.fence.resource.limited Maximum number of fence networks or ports
esx.problem.net.fence.switch.unavailable Switch fence property is not set
esx.problem.net.firewall.config.failed Firewall configuration operation failed. The changes were not applied.
esx.problem.net.firewall.port.hookfailed Adding port to Firewall failed.
esx.problem.net.gateway.set.failed Failed to set gateway
esx.problem.net.heap.belowthreshold Network memory pool threshold
esx.problem.net.lacp.lag.transition.down lag transition down
esx.problem.net.lacp.peer.noresponse No peer response
esx.problem.net.lacp.policy.incompatible Current teaming policy is incompatible
esx.problem.net.lacp.policy.linkstatus Current teaming policy is incompatible
esx.problem.net.lacp.uplink.blocked uplink is blocked
esx.problem.net.lacp.uplink.disconnected uplink is disconnected
esx.problem.net.lacp.uplink.fail.duplex uplink duplex mode is different
esx.problem.net.lacp.uplink.fail.speed uplink speed is different
esx.problem.net.lacp.uplink.inactive All uplinks must be active
esx.problem.net.lacp.uplink.transition.down uplink transition down
esx.problem.net.migrate.bindtovmk Invalid vmknic specified in /Migrate/Vmknic
esx.problem.net.migrate.unsupported.latency Unsupported vMotion network latency detected
esx.problem.net.portset.port.full Failed to apply for free ports
esx.problem.net.portset.port.vlan.invalidid Vlan ID of the port is invalid
esx.problem.net.proxyswitch.port.unavailable Virtual NIC connection to switch failed
esx.problem.net.redundancy.degraded Network Redundancy Degraded
esx.problem.net.redundancy.lost Lost Network Redundancy
esx.problem.net.uplink.mtu.failed Failed to set MTU on an uplink
esx.problem.net.vmknic.ip.duplicate A duplicate IP address was detected on a vmknic interface
esx.problem.net.vmnic.linkstate.down Link state down
esx.problem.net.vmnic.linkstate.flapping Link state unstable
esx.problem.net.vmnic.watchdog.reset Nic Watchdog Reset
esx.problem.ntpd.clock.correction.error NTP daemon stopped.  Time correction out of bounds.
esx.problem.pageretire.platform.retire.request Memory page retirement requested by platform firmware.
esx.problem.pageretire.selectedmpnthreshold.host.exceeded Number of host physical memory pages selected for retirement exceeds threshold.
esx.problem.scratch.partition.size.small Size of scratch partition is too small.
esx.problem.scratch.partition.unconfigured No scratch partition has been configured.
esx.problem.scsi.apd.event.descriptor.alloc.failed No memory to allocate APD Event
esx.problem.scsi.device.close.failed Scsi Device close failed.
esx.problem.scsi.device.detach.failed Device detach failed
esx.problem.scsi.device.filter.attach.failed Failed to attach filter to device.
esx.problem.scsi.device.io.bad.plugin.type Plugin trying to issue command to device does not have a valid storage plugin type.
esx.problem.scsi.device.io.inquiry.failed Failed to obtain INQUIRY data from the device
esx.problem.scsi.device.io.invalid.disk.qfull.value Scsi device queue parameters incorrectly set.
esx.problem.scsi.device.io.latency.high Scsi Device I/O Latency going high
esx.problem.scsi.device.io.qerr.change.config QErr cannot be changed on device. Please change it manually on the device if possible.
esx.problem.scsi.device.io.qerr.changed Scsi Device QErr setting changed
esx.problem.scsi.device.is.local.failed Plugin's isLocal entry point failed
esx.problem.scsi.device.is.pseudo.failed Plugin's isPseudo entry point failed
esx.problem.scsi.device.is.ssd.failed Plugin's isSSD entry point failed
esx.problem.scsi.device.limitreached Maximum number of storage devices
esx.problem.scsi.device.state.off Device has been turned off administratively.
esx.problem.scsi.device.state.permanentloss Device has been removed or is permanently inaccessible.
esx.problem.scsi.device.state.permanentloss.noopens Permanently inaccessible device has no more opens.
esx.problem.scsi.device.state.permanentloss.pluggedback Device has been plugged back in after being marked permanently inaccessible.
esx.problem.scsi.device.state.permanentloss.withreservationheld Device has been removed or is permanently inaccessible.
esx.problem.scsi.device.thinprov.atquota Thin Provisioned Device Nearing Capacity
esx.problem.scsi.scsipath.badpath.unreachpe vVol PE path going out of vVol-incapable adapter
esx.problem.scsi.scsipath.badpath.unsafepe Cannot safely determine vVol PE
esx.problem.scsi.scsipath.limitreached Maximum number of storage paths
esx.problem.scsi.unsupported.plugin.type Storage plugin of unsupported type tried to register.
esx.problem.storage.apd.start All paths are down
esx.problem.storage.apd.timeout All Paths Down timed out, I/Os will be fast failed
esx.problem.storage.connectivity.devicepor Frequent PowerOn Reset Unit Attention of Storage Path
esx.problem.storage.connectivity.lost Lost Storage Connectivity
esx.problem.storage.connectivity.pathpor Frequent PowerOn Reset Unit Attention of Storage Path
esx.problem.storage.connectivity.pathstatechanges Frequent State Changes of Storage Path
esx.problem.storage.iscsi.discovery.connect.error iSCSI discovery target login connection problem
esx.problem.storage.iscsi.discovery.login.error iSCSI Discovery target login error
esx.problem.storage.iscsi.isns.discovery.error iSCSI iSns Discovery error
esx.problem.storage.iscsi.target.connect.error iSCSI Target login connection problem
esx.problem.storage.iscsi.target.login.error iSCSI Target login error
esx.problem.storage.iscsi.target.permanently.lost iSCSI target permanently removed
esx.problem.storage.redundancy.degraded Degraded Storage Path Redundancy
esx.problem.storage.redundancy.lost Lost Storage Path Redundancy
esx.problem.syslog.config System logging is not configured.
esx.problem.syslog.nonpersistent System logs are stored on non-persistent storage.
esx.problem.vfat.filesystem.full.other A VFAT filesystem is full.
esx.problem.vfat.filesystem.full.scratch A VFAT filesystem, being used as the host's scratch partition, is full.
esx.problem.visorfs.failure An operation on the root filesystem has failed.
esx.problem.visorfs.inodetable.full The root filesystem's file table is full.
esx.problem.visorfs.ramdisk.full A ramdisk is full.
esx.problem.visorfs.ramdisk.inodetable.full A ramdisk's file table is full.
esx.problem.vm.kill.unexpected.fault.failure A VM could not fault in the a page. The VM is terminated as further progress is impossible.
esx.problem.vm.kill.unexpected.forcefulPageRetire A VM did not respond to swap actions and is forcefully powered off to prevent system instability.
esx.problem.vm.kill.unexpected.noSwapResponse A VM did not respond to swap actions and is forcefully powered off to prevent system instability.
esx.problem.vm.kill.unexpected.vmtrack A VM is allocating too many pages while system is critically low in free memory. It is forcefully terminated to prevent system instability.
esx.problem.vmfs.ats.support.lost Device Backing VMFS has lost ATS Support
esx.problem.vmfs.error.volume.is.locked VMFS Locked By Remote Host
esx.problem.vmfs.extent.offline Device backing an extent of a file system is offline.
esx.problem.vmfs.extent.online Device backing an extent of a file system came online
esx.problem.vmfs.heartbeat.recovered VMFS Volume Connectivity Restored
esx.problem.vmfs.heartbeat.timedout VMFS Volume Connectivity Degraded
esx.problem.vmfs.heartbeat.unrecoverable VMFS Volume Connectivity Lost
esx.problem.vmfs.journal.createfailed No Space To Create VMFS Journal
esx.problem.vmfs.lock.corruptondisk VMFS Lock Corruption Detected
esx.problem.vmfs.lock.corruptondisk.v2 VMFS Lock Corruption Detected
esx.problem.vmfs.nfs.mount.connect.failed Unable to connect to NFS server
esx.problem.vmfs.nfs.mount.limit.exceeded NFS has reached the maximum number of supported volumes
esx.problem.vmfs.nfs.server.disconnect Lost connection to NFS server
esx.problem.vmfs.nfs.server.restored Restored connection to NFS server
esx.problem.vmfs.resource.corruptondisk VMFS Resource Corruption Detected
esx.problem.vmsyslogd.remote.failure Remote logging host has become unreachable.
esx.problem.vmsyslogd.storage.failure Logging to storage has failed.
esx.problem.vmsyslogd.storage.logdir.invalid The configured log directory cannot be used.  The default directory will be used instead.
esx.problem.vmsyslogd.unexpected Log daemon has failed for an unexpected reason.
esx.problem.vpxa.core.dumped Vpxa crashed and a core file was created.
hbr.primary.AppQuiescedDeltaCompletedEvent Application consistent delta completed.
hbr.primary.ConnectionRestoredToHbrServerEvent Connection to VR Server restored.
hbr.primary.DeltaAbortedEvent Delta aborted.
hbr.primary.DeltaCompletedEvent Delta completed.
hbr.primary.DeltaStartedEvent Delta started.
hbr.primary.FSQuiescedDeltaCompletedEvent File system consistent delta completed.
hbr.primary.FSQuiescedSnapshot Application quiescing failed during replication.
hbr.primary.FailedToStartDeltaEvent Failed to start delta.
hbr.primary.FailedToStartSyncEvent Failed to start full sync.
hbr.primary.HostLicenseFailedEvent vSphere Replication is not licensed replication is disabled.
hbr.primary.InvalidDiskReplicationConfigurationEvent Disk replication configuration is invalid.
hbr.primary.InvalidVmReplicationConfigurationEvent Virtual machine replication configuration is invalid.
hbr.primary.NoConnectionToHbrServerEvent No connection to VR Server.
hbr.primary.NoProgressWithHbrServerEvent VR Server error: {[email protected]}
hbr.primary.QuiesceNotSupported Quiescing is not supported for this virtual machine.
hbr.primary.SyncCompletedEvent Full sync completed.
hbr.primary.SyncStartedEvent Full sync started.
hbr.primary.SystemPausedReplication System has paused replication.
hbr.primary.UnquiescedDeltaCompletedEvent Delta completed.
hbr.primary.UnquiescedSnapshot Unable to quiesce the guest.
hbr.primary.VmLicenseFailedEvent vSphere Replication is not licensed replication is disabled.
hbr.primary.VmReplicationConfigurationChangedEvent Replication configuration changed.
vim.event.LicenseDowngradedEvent License downgrade
vim.event.SystemSwapInaccessible System swap inaccessible
vim.event.UnsupportedHardwareVersionEvent This virtual machine uses hardware version {version} which is no longer supported. Upgrade is recommended.
vprob.net.connectivity.lost Lost Network Connectivity
vprob.net.e1000.tso6.notsupported No IPv6 TSO support
vprob.net.migrate.bindtovmk Invalid vmknic specified in /Migrate/Vmknic
vprob.net.proxyswitch.port.unavailable Virtual NIC connection to switch failed
vprob.net.redundancy.degraded Network Redundancy Degraded
vprob.net.redundancy.lost Lost Network Redundancy
vprob.scsi.device.thinprov.atquota Thin Provisioned Device Nearing Capacity
vprob.storage.connectivity.lost Lost Storage Connectivity
vprob.storage.redundancy.degraded Degraded Storage Path Redundancy
vprob.storage.redundancy.lost Lost Storage Path Redundancy
vprob.vmfs.error.volume.is.locked VMFS Locked By Remote Host
vprob.vmfs.extent.offline Device backing an extent of a file system is offline.
vprob.vmfs.extent.online Device backing an extent of a file system is online.
vprob.vmfs.heartbeat.recovered VMFS Volume Connectivity Restored
vprob.vmfs.heartbeat.timedout VMFS Volume Connectivity Degraded
vprob.vmfs.heartbeat.unrecoverable VMFS Volume Connectivity Lost
vprob.vmfs.journal.createfailed No Space To Create VMFS Journal
vprob.vmfs.lock.corruptondisk VMFS Lock Corruption Detected
vprob.vmfs.nfs.server.disconnect Lost connection to NFS server
vprob.vmfs.nfs.server.restored Restored connection to NFS server
vprob.vmfs.resource.corruptondisk VMFS Resource Corruption Detected
Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: ESXi, vSphere Tagged With: alarm, esxi, vob, vSphere

Handy VSAN VOBs for creating vCenter Alarms

04/22/2014 by William Lam 3 Comments

There have been quite a few questions lately around vCenter Server Alarms for VSAN, one in particular that I have noticed is around individual disk failure for VSAN. Outside of the generic default datastore alarms, there seems to be only two VSAN specific alarms:

vsan-default-alarms
I figure there must be other useful alarms that we could create, especially after showing how you can create a vCenter Server Alarm to monitor the VSAN component count threshold based on a particular VSAN VOB. I took a look around and found the following VSAN specific VOBs which could be useful for creating additional vCenter Alarms.

VOB ID VOB Description
esx.audit.vsan.clustering.enabled VSAN clustering services have been enabled.
esx.clear.vob.vsan.pdl.online VSAN device has come online.
esx.clear.vsan.clustering.enabled VSAN clustering services have now been enabled.
esx.clear.vsan.vsan.network.available VSAN now has at least one active network configuration.
esx.clear.vsan.vsan.vmknic.ready A previously reported vmknic now has a valid IP.
esx.problem.vob.vsan.lsom.componentthreshold VSAN Node: Near node component count limit.
esx.problem.vob.vsan.lsom.diskerror VSAN device is under permanent error.
esx.problem.vob.vsan.lsom.diskgrouplimit Failed to create a new disk group.
esx.problem.vob.vsan.lsom.disklimit Failed to add disk to disk group.
esx.problem.vob.vsan.pdl.offline VSAN device has gone offline.
esx.problem.vsan.clustering.disabled VSAN clustering services have been disabled.
esx.problem.vsan.lsom.congestionthreshold VSAN device Memory/SSD congestion has changed.
esx.problem.vsan.net.not.ready A vmknic added to VSAN network configuration doesn't have valid IP. Network is not ready.
esx.problem.vsan.net.redundancy.lost VSAN doesn't haven any redundancy in its network configuration.
esx.problem.vsan.net.redundancy.reduced VSAN is operating on reduced network redundancy.
esx.problem.vsan.no.network.connectivity VSAN doesn't have any networking configuration for use.
esx.problem.vsan.vmknic.not.ready A vmknic added to VSAN network configuration doesn't have valid IP. It will not be in use.

Looking at the list above, the following two VOBs seems like they would be useful for alerting on a disk failure is:

  • esx.problem.vob.vsan.lsom.diskerror
  • esx.problem.vob.vsan.pdl.offline

Disclaimer: There are no guarantees that a disk error or failure will automatically trigger these VOBs due to the unknown nature of how a disk may be fail, especially if it is intermittently.

Even though we can not simulate a disk error on a physical disk, we can still do some magic using a Nested VSAN environment. The worse case scenario that you could run into is that one of the disk just goes completely offline. We can simulate a similar behavior in a Nested ESXi environment by removing one of the virtual disks from the Virtual Machine (not deleting it).

To demonstrate the following scenario, here are the steps to create a vCenter Alarm for the following two VOBs:

Step 1 - Create a new vCenter Alarm and give it a name. Select “Hosts” for Monitor and “Specific event occurring …” for Monitor:

vsan-disk-failure-alarm-0
Step 2 - Add the following two VOBs above into the Event trigger:

vsan-disk-failure-alarm-1
Step 3 - Remove one of the Virtual Disks (SSD/MD) from the Virtual Machine running the Nested ESXi VM.

Step 4 - There are two ways in which you can trigger the alarm. You can either create a new Virtual Machine which will try to write to the Nested ESXi VM in which you remove the Virtual Disk or you can rescan the storage adapter for the Nested ESXi VM. In my environment, I happen to have a VM running on an NFS datastore and I performed a Storage vMotion of the VM onto my VSAN Datastore using the default FTT=1 policy on a three node VSAN Cluster. This immediately triggered the alarm as seen in the screenshots below:

vsan-disk-failure-alarm-2

vsan-disk-failure-alarm-3

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: VSAN, vSphere 5.5 Tagged With: alarm, esxi 5.5, vob, VSAN, vSphere 5.5

How to create vCenter Alarm to alert on ESXi 5.5u1 NFS APD issue?

04/19/2014 by William Lam 14 Comments

As some of you may have heard, there is currently a known issue with NFS based datastores (includes VSA NFS datastores) after upgrading to vSphere 5.5 Update 1. The issue causes NFS datastores to disconnect and go into an APD (All Paths Down) state. VMware is currently aware of the problem and you can follow KB 2076392 for the latest updates.

While going through my Twitter stream this morning, I noticed an interesting question from fellow Blogger and friend Jase McCarty who asked the following:

vsphere55u1-nfs-apd-alarm-2
I was quite surprised to hear that there were no vCenter Alarms being triggered for this issue. I decided to take a look at the KB to better understand the symptoms and see if there was anything I could do to help. From what I can tell, the only way to identify this particular problem is by looking at the logs which the KB has an example of what you would see.

Once I took a look at the logs, I knew there was at least two methods in which one could get alerts. One option would be to leverage vCenter Log Insight and create a query based on the particular string but no every customer is using Log Insight and it does require a bit of setup. The second more obvious option for me would be to key off of the VMkernel VOBs that are being generated which I have written about in the past for detecting duplicate IP Addresses for ESXi and VSAN component threshold count.

Here are the steps to create vCenter Alarm:

Step 1 - Create a new vCenter Alarm and give it a name. Select "Hosts" for Monitor and "Specific event occurring ..." for Monitor for

vsphere55u1-nfs-apd-alarm-0
Step 2 - For the Trigger, you will add the following VOB entries (just copy/paste them in)

  • esx.problem.storage.apd.start
  • esx.problem.vmfs.nfs.server.disconnect
  • esx.problem.storage.apd.timeout

Note: The alarm will activate if ANY of the VOBs are seen since it is an OR statement. It would have been nice to be able to group these together to generate the alarm

vsphere55u1-nfs-apd-alarm-1
Once the alarm has been created, you will at least have a way to get notified if you are potentially affected by this problem. I would still highly recommend you subscribe to KB 2076392 for all the latest updates.

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: ESXi, vSphere 5.5 Tagged With: apd, esxi 5.5, nfs, vob, vSphere 5.5

How to automatically monitor VSAN Component threshold using a vCenter Alarm?

04/14/2014 by William Lam 6 Comments

There was an interesting VMware KB article that was shared by Ron Oglesby last week which had caught my eye.

vsan-component-threshold
I had noticed earlier in the week that Ron was interested in finding the current VSAN Component count which is exposed in a variety of interfaces: RVC (vSphere Ruby Console) available on both Windows and Linux as well as through the vSphere API. I even created some recent scripts here and here using the vSphere API to remotely query the number of VSAN components for each ESXi host. I much prefer this option from a management standpoint and not have to log into each individual ESXi host.

After looking at VMware KB 2071379, I can see why Ron had asked his question as I also felt the KB was incomplete. However, to the unsuspecting eye it may not be obvious but the KB actually does contain the answer but it does not really go into any details that can be consumed by a customer. In the article, it mentions that VSAN has the ability to trigger an alarm when the threshold of the number of VSAN components on a particular host has reached 80%. What the article lacks are the details of how and where this alarm is triggered. First off, the alarm mentioned here is for vCenter Server. Secondly, this is made possible through the use of the VOB (VMkernel Observation) ID mentioned in the article. You can actually create vCenter alarms based on these ESXi host generated VOBs which I have written about in the past such as this one on detecting duplicate IP Address for your ESXi hosts. The process in creating this vCenter Alarm is pretty straight forward and I agree that this alarm should have been created by default (something I will raise internally with the engineering team).

Here are the steps to create a vCenter Server Alarm to notify at the 80% VSAN Component threshold:

Step 1 - Create a new vCenter Server Alarm and give it a name and select "Monitor specific event ..." for a Host and make sure it is enabled.

Screen Shot 2014-04-11 at 4.41.27 AM
Step 2 - Add in esx.problem.vob.vsan.lsom.componentthreshold for the Event

Screen Shot 2014-04-11 at 4.42.01 AM
Step 3 - You can leave the actions to be empty which will just generate a regular vSphere Alarm or you can specify an action.

Once we have our vCenter Alarm created, we will probably want to test and verify the alarm is working using either a Nested ESXi VSAN environment or an actual VSAN environment. The next question is how do we go about creating 2400 VSAN components? Well, instead of manually creating 2400 VMs which will probably take awhile, we can easily do so by leveraging a neat little utility found in the ESXi Shell called /usr/lib/vmware/osfs/bin/objtool

Disclaimer: The tools and scripts used in this article is mainly for education and information purposes The command below will create an object called object-1 with size 1KB leveraging the VSAN Policy of hostFailuresToTolerate=0 & forceProvisioning=1:

/usr/lib/vmware/osfs/bin/objtool create -s 1KB -a 3 -n object-1 -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"

For this particular test, we just want to quickly create 2400 VSAN components. To do so, you will need about 32GB of memory to reach the maximum amount of supported VSAN Components. This should not be a problem for a "real" VSAN environment but for my Nested ESXi environment I had to increase my resources for this test. Since VSAN is a Distributed Object Store, the objects being created will be randomly placed within a VSAN Cluster. To quickly get to 2400 components, I also put 2 out 3 ESXi hosts into Maintenance Mode to ensure all objects are created o the first ESXi host.

vsan-component-count-alarm-3
Finally, to assist with the creation of these VSAN Objects automatically, I created a quick script which you can run in the ESXi Shell

Shell
1
2
3
4
5
6
7
8
9
10
11
12
#!/bin/sh
 
START=1
END=2400
 
for i in $(seq ${START} ${END});
do
echo "Creating VSAN Component ${i} ..."
RES=$(/usr/lib/vmware/osfs/bin/objtool create -s 1KB -a 3 -n object-${i} -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))")
UUID=$(echo ${RES} | awk -F 'UUID:' '{print $2}')
echo ${UUID} >> /tmp/uuid
done

The creation of each object will have an associated UUID which will be saved into a temporary file /tmp/uuid and you can use the following script to delete the object once you have confirmed the vCenter Alarm works.

Shell
1
2
3
4
5
6
7
#!/bin/sh
 
for i in $(cat /tmp/uuid);
do
echo "Removing VSAN Object $i ..."
/usr/lib/vmware/osfs/bin/objtool delete -u $i
done

Once the 2400 VSAN Component count has been reach, you should now see the alarm we created earlier triggering for reaching the 80% threshold.

vsan-component-count-alarm-0

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: VSAN, vSphere 5.5 Tagged With: alarm, components, objtool, vob, VSAN, vSphere 5.5

Detecting A Duplicate IP Address For Your ESXi Hosts Using a vCenter Alarm

01/28/2013 by William Lam 5 Comments

The motivation for this article was a tweet I noticed from Duncan Epping this morning. Per Duncan's tweet, it looks like he may have accidentally assigned an IP Address to one of his virtual machines which was already being used by an existing ESXi host causing a duplicate IP Address error. We probably have all experienced this once in our lives and it can be quite difficult and frustrating to troubleshoot. Similar to a Windows OS, ESXi can also detect a duplicate IP Addresses but instead of a notification window, it is just logged in the VMkernel logs which looks like the following:

2013-01-21T15:52:35.989Z cpu1:2049)Tcpip_Vmk: 112: arp: 00:50:56:bd:3b:2b is using my IP address 172.30.0.213 on vmk1! 

The biggest challenge of course is to identify which ESXi host actually has a conflict and then taking a look at the logs to find the offending MAC Address and shutting them down yourself or with the help of a Network Administrator. Wouldn't it be great if we had an alarm to automatically notify us when a duplicate IP Address is detected? Well I am glad you asked and the answer is YES! 🙂

In addition to logging to the VMkernel logs, ESXi also logs this "observation" in /var/log/vobd.log which stands for the VMkernel Observation. These "observations" can provide critical identifying information in case of an error and is usually used during troubleshooting. In our case, we are seeing an intermittent network connectivity to our ESXi host which is in result of a duplicate IP Address. The really neat thing about these VOBs is that you can create vCenter Alarms when a specific VOB has been detected. I have shown an example of this before in my Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm article.

We can do exactly the same for detecting a duplicate IP Address for an ESXi host. The first thing we need to do is identify the VOB ID by looking in /var/log/vobd.log:

2013-01-21T15:02:07.513Z: [netCorrelator] 917174784727us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for 172.30.0.83 on interface vmk0, current owner being 00:50:56:bd:3b:2b

We can see the VOB ID for this is esx.problem.net.vmknic.ip.duplicate and this will be used in our vCenter Alarm trigger.

Step 1 - Create a new Alarm and specify a name, the Monitor type will be Hosts and Monitor For will be for a specific event:

Step 2 - Copy the VOB ID that we have identified from above and specify that as our alarm Trigger:

Step 3 - If you wish to receive an email notification or send an SNMP trap go ahead and configure additional actions, else just click next which will just display a vCenter Server alert in the UI.

Now that our alarm has been created, we will want to give this a test drive .... who can we ask? Well it just happens that I have a new user in my environment and I provisioned him a new VM which is already connected to the network. Let's hope he does not try to change the IP Address (because this never happens, right?)

After the user statically assigns the IP Address of an existing ESXi host in the VM, we should see our new alarm trigger in vCenter.

As you can see, we have quickly identified the ESXi host that is impacted and we can then login to DCUI via the console to take a look at the logs to find the offending MAC Address. Hopefully duplicate IP Addresses is not a common problem in your environment but it does happen from time to time and having an alarm to help you quickly narrow down the culprit can be quite useful.

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: Uncategorized Tagged With: alarm, duplicate IP, esxi, ip address, vob, vSphere

  • Go to page 1
  • Go to page 2
  • Go to Next Page »

Primary Sidebar

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Services Business Unit (CSBU) at VMware. He focuses on Automation, Integration and Operation for the VMware Cloud Software Defined Datacenters (SDDC)

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Sponsors

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy