Examples: Bio-inspired organizations and swarms are examples of robust organizations.

A Robust system should overcome these without much performance decrease.

• Outside Disturbances
• Internal Variation
• Agent Loss
• Faulty Information

Sensitivity to these abnormalities can be measured as the change in efficiency with respect to the performance criteria.

Additional measurements that should be low for robust systems.

• Time to Equilibrium
• Time to Local Consensus
• Overcommitment
• Wasted Resources
• Unproductive Effort
• Amount of State Thrashing

Examples: Hierarchical organizations are usually very responsive.

Things a hierarchical organization should have.

• Centralized Goal, Objective, or Task
• Information Passing
• Tasks, Roles, and Resources assigned to Agents

Ways to increase efficiency in hierarchical organizations.

• Maximize Information Gain During Communication
• Reduce Coordination Costs
• Reduce Social Loafing and Excess Idle Time
• Higher Level Decisions Affected by Information and Ideas from Lower Levels

From Sujit's Experiments.

• Responsiveness = Performance
• Robustness = Derivative of Performance w.r.t. Unreliability / The Unexpected

Robustness seems to need to be measured with respect to a type of unexpected phenomena. A few examples of unexpected phenomena are listed above. Since performance can be measure using a number of metrics, each performance metric can be tested for its robustness to a type of unexpected phenomena. The graph of the robustness of a performance metric w.r.t. the degree of an unexpected phenomenon will result in an unreliability inpact curve analogous to a neglect impact curve. From the board, we called this sensitivity. But perhaps robustness is the sum total of such sensitivity curves.

The word responsiveness seems like it should measure how well the system responds to some change, and not just system performance. Responsiveness could be measured as the time it takes to reach a type of organizational, informational, or environmental equilibrium after a task, role, goal, or organization change.

Perhaps performance should itself be a separate metric class.

From “Developing Performance Metrics for the Supervisory Control of Multiple Robots” by Jacob W. Crandall and M. L. Cummings. (HRI2007)

• Metric classes suggested in paper:
  • Interaction Efficiency
  • Neglect Efficiency
  • Attention Allocation Efficiency
• Necessities of metrics:
• Metrics should identify limits of all agents
• Metrics should have predictive power
• Metrics should contain key performance parameters that indicate overall effectiveness

From “Identifying Generalizable Metic Classes to Evaluate Human-Robot Teams” by P. Pina, M. L. Cummings, J. W. Crandall, and M. Della Penna. (HRI2008)

• Metric Classes suggested in paper:
  • Mission Effectiveness
  • Human Behavior Efficiency
  • Robot Behavior Efficiency
  • Human Behavior Cognitive Precursors
  • Human Behavior Physiological Precursors
  • Collaborative Metrics

The metric classes in the earlier paper listed seem to be represented in the latter.

• Interaction Efficiency < Human Behavior Efficiency
• Neglect Efficiency < Robot Behavior Efficiency
• Attention Allocation Efficiency < Collaborative Metrics

It seems correct to split the behavior of the robot and human into different metrics. Collaborative Metrics could also be split into human-human, human-robot, and robot-robot situations. This could help identify (or predict) the location of a problem, whether it is a human problem (“problem exists between keyboard and chair”), an autonomy problem, or an organizational problem.

“Entropy of activity, entropy of information, .. ” “The idea is that entropy should have a time-varying quality that satisfies some pattern.” “I'm not sure what that pattern should be.”

A few possible types of entropy for metrics:

• Behavioral Entropy
• Informational Entropy
• Organizational Entropy
• Task-Ability Entropy? (Sort of a measure of how well the agents are suited for the types of tasks in the mission. Based on affordances of agents, requirements of the tasks, and the environment)

What types and measurements of entropy would be useful in these situations?

Informational entropy over time could be a good way of measuring information flow and how information is distributed among agents.

Behavioral entropy could be a good way of measuring a human's state (from driving and measuring how sleepy people are).

What would organizational entropy be? Would it require organizations to self-organize?