all words are now rare

baseline (inspired by bug): train model at round 1 (on batch 1) and use it across the experiment - essentially longest sentence

alternative: stop updating model at round x - (similar to switching to random selection at round x) - for some sufficiently large x, should see no disadvantage

issue: cost of waiting for computer to select next sample

idea: time-limited sample selection (use stale scores, if time doesn't permit more work) - time limit could be fixed const. or determined by completion of annotator's work unit.

measure cumulative time of experiments

idea: most common words per length of sentence

oracle labeled data split into training and test.

• Multiple annotators/annotations
• Simple model w/better selection (at first)
• More look-ahead w/ simple model
• Interactive constrained Viterbi (cost perspective)
• 3 processes
• Proper initialization of prior – convergence of fast maxent (acero & Chelba)
• QBC:
  • Committee selection
  • Disagreement metric
  • Size
• Randomize of unannotated set every round
• Profile
• Net cost = gross cost – annotation time
• For 1 cycle of a.l.
  • A. both idle (have this)
  • Human idle
  • No-one idle
  • Computer idle
• Candidate set size
• Data/results “commit”

Q_HC Q_CH 0 0 \infty 0 \infty \infty 0 \infty

• First two rows: pay when human waits; no pay when human waits

Diagram: edge from H to C and from C to H. Edges annotated by Q_HC and Q_CH, respectively

Questions:

• 1 character per prefix? yes
• more than one prefix? 0-3
• is a single separator ambiguous in some cases? use 2: one for prefix and one for suffix
• multi-purpose interface: active learning, browsing, review for proofing
• are we tagging prefixes or just segmenting them?

Action: Brandon: clicking on side words updates attribute-value box's label Peter: the prefix string is a value tagged by the Syriac tagger. shouldn't be. Eric: follow up with Harry Diakoff

• Auto-complete in little-language box
• Support keyboard-only data entry
• Update public site for ALFA project

Prefixes:

• dolath (d)
• lamad (l)
• waw (conj)
• prepositions

Suffixes: small, finite number

Idea: Layered / Prioritized Tags

• tool should support full expressivity (full tag set)
• tool should be configurable to hide layers (reduced tag set)
• Configurable: may later want other layers of annotation, other distinctions

Important: supporting linkage of stems to headwords (“lexemes”) in the dictionary

Modes:

• Active learning
• Review

Features:

• Reveal context: tool-tips should reveal attributes on more distant tokens in any text view
• configureable amount of context
• situated in the corpus, in a document
• ability to browse files: explorer right on the left
• ability to browse dictionary: dictionary pane on the right
• ability to link stems to lexemes in the dictionary
• ability to add to dictionary with pointer(s) back to corpus for examples

Roles for annotators:

• editors of dictionary
• non-editors - can only propose new entries

Questions / variables for user study:

• scope of jumps taken by active learner / measure the cost of context switching
  • article
  • genre
  • time period
  • author
  • corpus
• availability of lexicon
• how much context
  • sentence, phrase, QWIC, etc.
• granularity of annotation:
  • sentence
  • phrase
  • word
  • word sub-tag
• correct or annotated from scratch
• presentation of top-N model hypotheses
• order of forced annotation: step-by-step in order or jumping (per AL)

Research question:

• 1st item choice
• online cost model estimation
• which model to select data for user study
  • minimize bias of item selector for user study
• layers of annotation

Separate problem: vowel restoration

Action:

• Hebrew OOV investigation

Interface modes: 1. active learning **machine determines granularity: sentence, phrase, word, sub-tag 2. review mode: sequential order 3. review mode: arbitrary order 4. review mode: AL order 5. browse (no changes)

Projects to complete:

• User study for Syriac
• Simulating AL for Syriac
• Cost model on the fly
• Get Habash/Rambow data for Arabic

Proposal to Harry Diakoff

• Machine learning
• Annotation
• could be joint with BYU Classics or with Perseus Project
• 2 pages

Paper ideas:

• Wait for it! Cost/benefit trade-offs in waiting
• probability of datum - later (15 July 08 ?) decided to be unpromising based on prior experiments and discussion
• utility / loss as part of active learner
• multiple annotators / imperfect annotators
• Cost model on the fly
• Cost implications of error correction propagation
• see Culotta & McCallum
• another point on the correction vs. from scratch spectrum
• Greedy EVSI
• Particle filters for Bayesian models in AL

Ask Ivan @ MS about Win Server licenses

Re-engage with Marc Carmen

Paul & Brandon do web-based prototype

• Paul: GWT & JSF
• Brandon: ASP & JSF

Eric: architecture for client/server set-up in Visio diagram

Features for prototype

• Inspect lexical entry
• Nestorian font
• Font size

Features for review mode in prototype

• highlight word in top line
• single line of review cells
• divider bar separating text (top) from review cells (bottom); ability to move to create more or fewer lines of review cells
• allowance for multiple annotations
• ability to change
• indicator: tagged by human (blue) or machine (yellow)
• reveal auto tag. on demand
• reviewer can acknowledge blue and yellow tags
  • turns green
  • retrains model, as appropriate
• reveal levels of confidence on yellow cells - little confidence bars?
• editing lens
• progress tracker

Features for active learning mode in prototype:

• “next” button should be “done button” when word-at-a-time
• sent.-at-a-time: previous word, next word, done
• “back” button to return to previous case
• highlight
• hide file browser
• remove review boxes above and below the “lens” row
• add path (corpus –> author –> doc) at top of view for context
• place the highlighted word in the middle
• use yellow & blue highlights on text in text view and in edit controls
• constraint and prediction
• changing attribute affects lexeme and vice versa
• segment, then constraint lexicon, then attributes

Features for browse mode in prototype

• tool tips on every word
• link to dictionary

Features for all modes in prototype

• annotator should be able to flag a transcription as possibly erroneous
• allow viewing of image?

Focus: knowledge-free

Perspective: cost reduction

• compare machine learning (data-driven approach) with knowledge engineering

Partial solutions:

• morph. tagging (as we have defined it - one vector of attributes per token)
• segmentation only
• vowel restoration

Whole solution:

• tag + segment
• look up in / link to dictionary
• required for Syriac project

Reminders on Syriac tagger:

• Re-do without string prediction
• Remove vowels and re-do

Proposals:

• feature sets for predictive segmentation

1. predict # of characters in prefix, suffix 2. choose among letter sequence as prefix