Research Programme · Human-Computer Interaction
The peer-reviewed FingerReader2.0 paper (IMWUT 2018) presents a wearable finger-worn camera system designed to help people with visual impairments identify products and bank notes during real-world shopping tasks.
Featured Publication
The paper addresses practical shopping barriers for PVI, especially product identification in supermarkets where labels are small, similar products are hard to distinguish, and phone framing is difficult without sight.
Instead of relying on handheld capture, FingerReader2.0 uses natural pointing and a single-thumb tap ("point-and-shoot"), while keeping hands free for cane, basket, and object handling.
The team conducted interviews, observations, focus groups, and iterative sessions with visually impaired participants. Across the design lifecycle, 26 participants contributed to requirements, interaction design, and usability refinements.
One key design result was simplification: removing complex ring gesture menus and retaining a single tap trigger reduced cognitive load and improved practical use.
Finger-worn ring with embedded camera and touch interface for single-tap capture. Adjustable fittings were used to improve fit and pointing stability.
A wristband houses processing electronics and battery, enabling wearable use without tethering to a desktop setup.
Speech feedback is synthesized on phone and delivered through phone speaker or Bluetooth headset, supporting fast recognition-response loops.
On-board SSD MobileNet recognition prioritizes speed for known classes; cloud vision fallback provides scene/text support when confidence is low.
Field Study Summary
| Study Component | Method | Key Finding |
|---|---|---|
| Participants | 5 right-handed PVI in supermarket field study | Included varied visual conditions and prior assistive-tech habits; all had prior device exposure from earlier iterations |
| Tasks | Item/product identification and cashier payment flow | 236 recognition attempts across 17 items (products + bank notes) captured in natural shopping context |
| Recognition mix | On-board SSD + cloud vision fallback | On-board model delivered most successful identifications; cloud fallback added broader but slower descriptive support |
| Latency | Measured request/response timing | On-board processing was substantially faster (about 1-2s typical) than cloud fallback (often above 3s) |
| Usability | SUS questionnaire + video analysis | Average SUS was 67.5, with strong dependence on pointing technique, framing, distance control, and hand steadiness |
Design Insights and Limits
Pointing quality drives performance. Successful use depends on aiming the object center, keeping appropriate distance, minimizing occlusion, and waiting for capture feedback before movement.
Simplicity beats feature overload. The study found that a single-tap interaction outperformed richer gesture menus for many users in real tasks.
Environmental constraints matter. Glare, low light, reflective packaging, and object orientation significantly affect recognition reliability.
Current limitations. The paper notes constraints in scalability (class expansion and annotation effort), dataset diversity, ergonomics, and robust handling of difficult framing conditions.
Future direction. Expanding detectable products, improving guidance for framing/orientation, and reducing computational constraints are identified as major next steps.