Greg Stanley

• Managing diverse networks

• Major operational goals

• Major components

• Alarm filtering & correlation examples

• Numerous device types & manufacturers

• Data vs. real-time for voice & video

• Different protocols (TCP/IP, CMIP, ...)

• LAN/WAN differences

• Wireless vs. terrestrial

• Changing topologies

• Complex devices

• Subsystem interfaces & proxies

Element management systems

• Flexibility

• Speed of development - overall development environment

• Incremental development environment for rapid development & feedback, partial solutions

• Representation power: modelling the systems for use in diagnostics, analysis, prediction, ...

• Portability between platforms

• Systems integration capabilities

• Early detection of problems (proactive)

• Alarm/message/event filtering

Suppression of repetitive alarms

• Alarm correlation

Grouping of related alarms

• Diagnosis

Pinpointing the causes of alarms

• Procedure automation

Testing for diagnostic & filtering purposes

Resolving problems

Enforcing standard procedures

• Online information

Help, topology, hierarchy, relations

• "What-if" simulations for analysis or training

• Alarm X occurs, then clears by itself within timeout. Suppress it (do not present to operator).

• Alarm X occurs. Further testing reveals this alarm to be false or to have cleared itself. Suppress it.

• Alarm X is repeated n times. Present first alarm only, update a repetition counter

• Alarm X is not a real problem until it occurs n times within timeout. Present one alarm only, after n alarms, update repetition counter

• Alarm X and Alarm Y occur within timeout. Suppress these, present new message Z to operator

• Alarms X1, X2, ..., X6, sent from different agents, are all complaining about "target" device Y. Acknowledge X1...X6, and send an alarm about Y.

• Alarms X1, X2, ..., X8 were all sent by "sender" device Y. Send an alarm indicating suspicious behavior of Y.

• Multiple devices X1, X2, ... are sending messages complaining that they cannot communicate with device Y. Send a message that device Y has failed, and acknowledge all the messages for X1, X2, ...

• High-level services requiring particular interface cards X1, X2, ... are all failing. X1, X2, ..., are all plugged into a common backplane or have some other common failure mode. Diagnose and alarm on the common mode failure, and acknowledge X1, X2, ... .

• Alarm X occurs. Wait 60 seconds. Check for symptoms again by polling, or log in to a computer execute some UNIX commands (using remote shell). If the problem is still there, send an alarm, otherwise suppress it (except for an optional log entry).

• Knowledge management view

• "Build yourself a graphical language" to more closely match your tool to your domain

• Representation in OPA

• Emphasizes representation of knowledge for applications

• Not just data!

- System hardware & software models

- System topology

- Failure & fault propagation models

- Operating rules & procedures

• Example: alarm correlation & diagnostics need process topology and device models, also usable in operator training and in planning.

• Object-oriented, with graphical representation

• KBES represents both qualitative and quantitative models

• Object orientation is the key part of modern expert systems

• KBES represent information explicitly, rather than embedded in code

• Emphasis on building "declarative" descriptions, independent of subsequent use, and easily inspectable by wide class of users

- Goal to simplify representation & re-use of knowledge for multiple purposes

• Some KBES's (G2) have strong graphics orientation as part of its declarative knowledge

• Static vs. real-time KBES

• Development environment

• High-level descriptions:

- Equipment class implies behavior

- Schematic drawings: connections imply fault propagation, data flow, reachability, reliability

- "Part-of" relation implies fault propagation model

- "Is-a-kind-of" specialization simplifies descriptions

- Generic statements utilize these high-level constructs to generate specific diagnosis or simulation, using common attributes

• Model declarations are independent of ultimate usage

• Qualitative models (e.g., cause-effect)

For any telecom-device D connected to any hub H

if the status of H is failed

then conclude that the status of D is failed

and conclude that the alarm-priority of D = the alarm-priority of H

For any electrical-device D

whenever any message MSG becomes an-event-for D

and when the count of each message MSG2 that is an-event-for D > 4

then conclude that ....

and start multiple-message-filter(D)

• Reduces gaps between system analysis, specification, design, implementation, run-time use, maintenance.

- Explicit models carried through all phases

- Inspectable by all classes of users, not just programmers

• Common representation for multiple applications, with one consistent model for development & maintenance

• Generic library: default behavior specified for given class of object, connections - no additional special lists to fill out unless object deviates from the defaults

• Objects with attributes

• Class hierarchy for objects, with inheritance of properties and behavior - allowing "differential modelling"

• Associative knowledge, relating objects in the form of connections and relations

• Structural knowledge (e.g., "part-of" relation)

• Representation and manipulation of objects and connections graphically

• Generic rules and associated inference engine

• Concurrent procedures

• Analytic knowledge, such as functions, formulas, differential equation simulation

• Real-time task scheduler, supporting concurrency, priorities, time stamping, validity intervals, timed actions, event-driven activity, reasoning within a fixed deadline, history-keeping, data interfaces

• Interactive development environment and run-time environment

• Graphics

• External interfaces for systems integration

• Match tool to domain - reduce semantic gap between tool and problem

• Build library of classes & methods (procedures), rules, etc.

• Build "configurer" GUI based on cloning objects from a palette, connecting them, filling out tables of attributes

• Fairly common in many domains

• Containment hierarchy/"part-of" for physical areas, common-modes, physical equipment, hierarchy

• Objects in a class hierarchy with specialization & inheritance.

• Objects include attributes and methods (procedures), e.g., test methods

• Almost everything, whether physical or abstract, is an object

• Graphical connections represent physical connectivity, logical connectivity, or relationships such as cause/effect, hierarchy

• Logic networks (AND/OR gates, etc.)

• Fault trees, decision trees, AND/OR trees, hierarchical fault models, with goal-seeking

• Cause/effect diagrams

• Procedures to analyze schematic/map

• Telecom devices and software processes

• Class hierarchy

• Workspace ("part-of", "containment") hierarchy

• Containers ("sites", "networks")

• Alarms/messages/events

• Relations

• Connections - topology information

• Test and operator actions representation

• Common framework shared between message-handling, OPAC graphics language, and schematics

• Decode messages as needed, including identification of target, sender, category

• Eliminate obvious repetitions by simple message filtering

• Create "raw" warning messages

• Apply model-based diagnosis, heuristics, procedural reasoning when possible

• Acquire additional information & run tests

• Select candidate "most likely" failures based on model or other information

• Draw conclusions about root causes and sympathy events, prove nodes "good" or "bad"

• Cluster remaining alarms into reasonable groups when possible

• Automatically fix problems where possible

• Notify the operator with summarized alarms and other alarms, guide through repairs

• Pass information to trouble ticket system

• Recognizing recurring problems & notify system administrator

• Overall Architecture/System Integration

• Major components

• Messages are objects with attributes such as priority, acknowledgement status, time stamp, timeouts for proecedure execution such as escalation, etc.

• Message handlers store messages

• Individual views or messages handlers can be set up per Telewindows user

• Messages are organized and related to "target", "sender", "ID" (category), window, etc., for analysis or browsing.

• Unified framework with OPAC, e.g., same "target", "sender", "ID" (OPAC uses"notify" to designate message-handler

• Messages determine the priority and acknowledgement status of objects in the schematic (map)

• Programmatic access, as well as access by users

• Acknowledgement

• Deletion

• Optional modification (e.g., comments)

• Navigation to find sender, target, etc.

• Navigation from schematic objects to browse messages at any level (object, or larger unit with a subworkspace hierarchy)

• GSI C-based library to build custom bridges

• SQL-type interfaces to databases (Oracle, Sybase, ...)

• OpenView (SNMP/DM) interface

• File I/O and process spawning

• Interfaces G2/OPA to OpenView and general network via SNMP

• Runs as separate process, on same CPU as OpenView (G2 generally runs on a different machine)

• Written in C, using binaries from GSI library and OpenView library

• Works with OpenView DM platform, or the SNMP platform (which is a subset of DM)

• Supports standard SNMP get, get-next, set, send-trap, receive-trap

• With DM platform, can register for events using HP Event Management Services

• XMP calls (which support CMIP protocol) available later

• "Blocking" and "Non-blocking" modes

• G2/OPA can initiate interactions, or receive unsolicited traps

• G2/OPA can poll, or do management by exception

• G2/OPA can communicate directly with other managers, agents, or software (e.g., Bridgeway's Eventix) by getting and sending traps

• G2/OPA can change colors on OpenView map by sending standard "status" trap

• Operators using OpenView Windows can send traps directly to G2 (via "snmptrap" utility, after configuration of executable icons or menu entries)

• G2/OPA communications to or from bridge are G2 remote procedure calls

• Might configure G2 as an "intelligent operator", or as an "intermediary", intercepting all alarms on the way to OpenView

• Might use polling or management by exception

• Large distributed system may require some filtering, parsing & tokenization of alarm messages close to the sources

• Typical need to launch UNIX processes, receive results, read and write files

• OPAC language blocks directly execute spawns, file I/O

• AT&T EasyLink (Commercial electronic mail service): Using OPA for alarm filtering & diagnostics, procedure automation

• Intelsat: network monitoring, satellite telemetry monitoring

• Stanford Telecom ATM applications: DoD SSCN & SPANet, ANMA ATM manager

• Texaco Trading & Transportation

• SWIFT (Belgium) - monitoring bank wire transfers

• CRT Banca (Italy), others: Remote bank security monitoring

• Telefonica (Spain)