fSense

User wearing smartwatch with fSense and grasping a tennis ball User wearing smartwatch with fSense and grasping a tennis ball

Overview

fSense recognises the forces exerted during gestures using photoplethysmogram (PPG) or heart-rate sensors in smartwatches.

It can detect two levels of forces exerted (Soft or Hard) across 14 different gestures and grasps (see figure below) via a Random Forest classifier.

Two rows, 7 images per row. Top row, 1. thumb-tap. 2. normal fist. 3. Pinch. 4. Power sphere, grasping a spherical object. 5. Medium wrap, similar to a racket grip. 6. Tripod grip, two fingers and thumb on a ping-pong ball. 7. Lateral pinch. Bottom row, image number 8. Watch-tap. 9. Precision Disk, similar to holding a jar lid. 10. Lateral Tripod, opening a wine bottle. 11. Light Tool, gripping a pencil. 12. Index-extend, gripping a pencil with index finger extended to the tip of pencil. 13. thumb-2, holding pencil with two fingers and thumb. 14. Thumb-3, holding pencil with three fingers and thumb.
Gestures in which force can be detected using fSense

I assisted the project lead, Thisum Buddhika, in conducting a user study to collect the PPG data and building the classifier. Our work resulted in a conference paper in Augmented Human 2019.

Process

Data Collection

PPG data was collected from 12 users who were asked to wear a Samsung Galaxy Gear 3 Frontier smartwatch. They performed the 14 gestures at 4 force levels (No force / Soft / Medium / Hard) for 3 times per gesture. Thus, each user performed a total of 14 x 4 x 3 trials = 168 trials. Each gesture took 1 to 2 seconds to perform. The session was about 25 minutes.

Signal Processing and Feature Extraction

The PPG data was smoothed with an exponential moving average filter and the gradients of the signal were calculated. Statistical features of the mean, SD, min, max, median, skewness, kurtosis and RMS were extracted from the processed data in a sliding window of 20 data points (50% overlap).

Classifier

Three classification algorithms were compared: Support Vector Machine (SVM), Multinomial Logistic Regression (MLR) and Random Forest (RF). Initially, we intended to build a 3-class (Soft / Medium / Hard) force level classification system but our evaluation showed that it was more feasible to build 2-class systems (Hard+Medium/Soft, Hard/Medium+Soft).

The figure below summarises the classification accuracies and root mean squared error (RMSE) values from the evaluation. The RF algorithm was selected as it achieved higher accuracy and lowest RMSE compared to SVM and MLR.

A table showing classification accuracy and RMSE values of the algorithms for the 3-class and 2-class problems. Accuracy in percentage for 3-class (hard, medium and soft) problem for MLR is 42.3%, RMSE is 0.4667. Accuracy for SVM is 38.7%, RMSE is 0.5209. Accuracy for RF is 46.3%, RMSE is 0.4571. For the 2-class problem where one class is to classify Hard and Medium force and the other class is Soft force level, accuracy for MLR is 67.4%, RMSE is 0.4588. Accuracy for SVM is 67.4%, RMSE is 0.5705. Accuracy for RF is 71.7%, RMSE is 0.4451. For the 2-class problem where one class is to Hard force level and the other is Medium and Soft force levels, accuracy for MLR is 67.2%, RMSE is 0.4657. Accuracy for SVM is 67%, RMSE is 0.5746. Accuracy for RF is 69%, RMSE is 0.4515.
Accuracy comparison of classification algorithms for different force level combinations (H: Hard, M: Medium and S: Soft)