TiLAR v2 brainstorming notes

We can use Troy's face as a visual feedback channel. Instead of displaying text or something else typical, we can make it more graphical. For example, his eyes could change into beach balls for the ball-pushing game, or popcorn balls to sing popcorn popping.

MAG: Such an approach would free us from having to develop a standard laptop-based GUI for the therapist. Instead, the therapist could cycle through activities, with the eyes reflecting the robot's intent.

MAG: The eyes needn't change into beach balls. Instead, the eyes could get bigger with the center of the eye becoming a beach ball. This could explicitly represent the robot's intent.

MAG: Is this a way to make the “theory of the robot's mind” accessible to a child too? It might help the therapist understand the robot's state, but it might also help the child understand intentional behavior and theory of mind concepts.

MAG: possible user study I picture a three by two experiment.

• Three feedback modes: haptic, conventional laptop-based GUI, and theory of mind visual feedback.
• Two levels of task difficulty: simple set of possible robot states under low secondary task load, simple set of possible robot states under high secondary task load.

MBC: We may be able to design the vibrotactile “haptic icons” to feel like the activities that they are trying to represent. For example, Wii remote rumble that feels like a ball bouncing. We'll have to explore whether this is feasible.

TM: Would changing Troy's face be distracting to the child?

The Wii remote has excellent IR point tracking. We can use this for gesture recognition. If we put some (maybe even just 1) IR LED somewhere on Troy, then the therapist can point the remote at Troy, pull and hold the trigger, then wave it around like Harry Potter casting a spell. The Wii remote also has accelerometers, so we've got probably more features than we need for good gesture recognition. I know I have seen work with people using the Wii remote for gestures, since that's basically what it's for, so there may even be some existing code for gesture recognition specifically for the Wii remote.

MAG: This might be an easy way to test the feasibility and usefulness of the hand gesture recognition work that Honda is working on before this work is completed.

TAI Planning

Version 2 as last modified by Robert Brown August 2010.

The TiLAR UI is built on a generic framework that can be extended as needed. The framework has three main components: Nodes, Ports, and Arrows. These can each be subclassed for specific behavior. A Node contains two different types of Ports – InputPorts and OutputPorts. OutputPorts are connected to InputPorts by an Arrow, and vice versa. Arrows are simply for managing the connections between Ports and visually showing these connections.

In addition to these basic components, there are some other classes present for convenience. Node is subclassed into Commands and Controllers. Both are only partially implemented and currently aren't being used. With this layout, it is intended that Nodes represent actions that a robot can perform and a controller to drive those actions. For the therapist driven design model, it is probably best to subclass Ports as actions rather than Arrows.

Nodes are deleted by pressing the red X button in the top right corner. Arrows are deleted by dragging from one of the connected Ports and releasing in an empty area.

Connections are made between Ports by dragging from the one Port to another. Inputs cannot be connected to inputs and outputs cannot be connected to outputs. The connections are drawn as cubic bezier paths. Straight or orthogonal lines may be implemented if preferred. Currently, Arrows have no arrow head drawn, but direction is implied. The direction goes from outputs to inputs as should be expected. However, connections do not have to be made in this direction. A user may drag from an InputPort to an OutputPort.

Only the UI portion of the code has been implemented. The underlying logic needs to be added.

The ActionLibrary needs to be populated with the various commands available. The commands are organized in a QTreeView. This tree view is populated with data from a QStandardItemModel. The current structure in mind is to have controllers and robots at the top level. Underneath the controllers will be the various controllers available, such as a Wii Remote. Underneath the robots will be positive, negative, and neutral sections or something of the sort. Below these sections will be the categorized actions. It would also be beneficial to have favorite and recent actions sections. Once this is fully implemented, the generic add Node and add Wii Remote buttons should be removed from the toolbar.

Once the ActionLibrary is populated, double clicking on an item or clicking “add selected node” should result in the action being added to the workspace.

When the ActionLibrary is finished, the 3D robot animations may be incorporated into the UI.

There is currently no method for requesting what Port/Node a Port is connected to. This will be necessary.

It would be nice to be able to change a Node rather than delete it and replace it with another. This will require implementing a way to selection one, possibly more, Nodes. For single Node selection, this can be handled by the Node's mouse events. For multiple Node selection, this would need to be implemented in the QGraphicsView by overriding the mouse events. QGraphicsView already has a rubber band selection mode available. If this event emits a signal, then this may be a better choice than subclassing QGraphicsView. Currently the hand drag mode is being used to scroll the view. It may be beneficial to be able to switch between rubber band selection mode and hand drag mode.

Zooming in and out with the mouse wheel may be beneficial.

If desired, Nodes, Ports, and Arrows can have colors. They can also have different line styles. The line styles aren't fully implemented yet. An example of what can be done with this is to have different paths from a Controller be different colors and styles. It would be best to have one color associated with each line style for color blind users if this method is used.

The current memory management appears to get most, hopefully all, leaks but there may be some missed.

With how the design has changed over time, Arrows may be entirely unnecessary. The Arrow property of a Port could be replaced with a pointer to the connected Port. This could greatly simplify the code, especially with the memory management. This also means that Port would be responsible for drawing the connection between Ports.

The therapists would likely benefit from a Node that looks like the actual controller. (See the following section for instructions to make such a Controller.

For each controller with a custom image, subclass Controller. Next, override the boundingRect() and paint() methods. These will specifically define how the Controller is drawn in the view. Also override the type() methods for type handling. In the constructor, create a ButtonPort for each button on the remote. Be sure to set their position to line up with the buttons on the image. ButtonPorts are a subclass of OutputPort. They have no visual image. The purpose of a ButtonPort is simply for hit-testing. If you desire some kind of visual effects, such as highlighting the button, subclass ButtonPort to include this behavior.

Qt Creator may have problems with running the same code on different machines. The file paths may be stored as absolute paths rather than relative paths.

Nodes, and the other main components, can store arbitrary data. This feature is inherited from QGraphicsItem. The stored objects should just need to inherit from QVariant. They can be stored/fetched by a key of type int. See QGraphicsItem for more information.

When removing Nodes, Arrows, and such, there is a precise order of events that must occur. If everything is not deleted in the proper order, then memory leaks and/or segmentation faults will occur. For example, when a Node is deleted, its Ports must be deleted. This also means that any arrows connecting the deleted Ports must be deleted. When these Arrows are deleted, the Ports connected to the deleted Ports must have their Arrow pointer nulled. Looking at the TAI data model will help in understanding the many relationships between Nodes, Ports, and Arrows.

It is assumed in the current model that there is a one-to-one relationship between Ports. This means that one Port cannot be connected to many Ports. This behavior may be changed, but shouldn't be necessary. If a Node must connect to multiple Nodes, simply create multiple OutputPorts. Since this behavior is being used, anytime a connection is changed the old connection(s) that violate this one-to-one relationship will be deleted.

The TAI subversion repository also contains several design documents. Access to the TAI repository is required to view these documents. Wiki pages do not allow these documents to be uploaded.