TinyML 惯性手势识别与相机控制TinyML · Inertial Gesture Camera Control
使用 ESP32 与 MPU6050 采集惯性手势，完成轻量模型训练与端侧推理，将识别结果映射为相机指令；研究成果发表于 IJET（2025）。ESP32/MPU6050 gesture acquisition, lightweight model training and on-device inference, mapped to camera commands. Published in IJET in 2025.
项目背景Project Overview
将惯性数据采集、模型训练、嵌入式推理与相机执行组织为一个系统。训练模式把手势数据传到主机；识别模式在 ESP32 上运行轻量模型，由相机执行端解析控制指令。The system connects inertial-data acquisition, model training, embedded inference and camera execution. Training mode transfers gesture data to a host; recognition mode runs a lightweight model on the ESP32 and sends commands to the camera.
技术目标Engineering Objective
在嵌入式计算与内存约束下，识别动态手势并触发明确的相机动作。关键工作是保持训练和推理的输入处理一致，并把分类结果连接到可检查的指令协议。Recognize dynamic gestures under embedded compute and memory constraints and map them to explicit camera actions. The engineering task is to keep training and inference preprocessing consistent and connect classification to an inspectable command protocol.
系统架构System Architecture
- 01采集：ESP32 经 I²C 读取 MPU6050 加速度和角速度。Acquisition: ESP32 reads MPU6050 acceleration and angular velocity over I²C.
- 02训练：Wi-Fi UDP 传输数据到主机，整理时序输入并训练 TensorFlow/Keras 模型。Training: transfer data to a host over Wi-Fi UDP, prepare time-series inputs and train a TensorFlow/Keras model.
- 03推理：转换为 TensorFlow Lite，在 ESP32 上输出手势类别概率。Inference: convert to TensorFlow Lite and produce gesture-class probabilities on ESP32.
- 04执行：串口指令映射到 MaixCam 拍照、录像或回放操作。Execution: map serial commands to MaixCam capture, recording or playback.
关键技术实现Technical Implementation
IMU 数据与双模式流程IMU data and operating modes
论文描述了加速度/角速度的平滑、归一化与固定窗口采集。训练模式保存按手势分类的数据，识别模式重复同类预处理，再将窗口送入轻量模型。The paper describes smoothing, normalization and fixed-window acquisition of accelerometer and gyroscope signals. Training mode stores labeled gestures; recognition mode applies corresponding preprocessing before inference.
模型训练与部署Model training and deployment
在主机端使用 TensorFlow/Keras 训练轻量全连接分类模型，经模型转换后由 TensorFlow Lite 在 ESP32 上推理。论文中的三类手势为 cross、circle 和 w。A lightweight fully connected classifier is trained with TensorFlow/Keras on the host, converted and run with TensorFlow Lite on ESP32. The paper studies three gesture classes: cross, circle and w.
相机控制协议Camera-control protocol
识别结果映射为 m_a、m_b、m_c 一类串口命令。相机侧按起始字符、分隔符和动作字符逐字节解析，连接分类输出与拍照、录像和回放操作。Recognized gestures map to serial commands such as m_a, m_b and m_c. The camera parser processes a start character, separator and action character to connect classification with capture, recording and playback.
实验评估Experimental evaluation
论文分别讨论分类、实时交互和通信实验。项目讲解将离线分类、实时识别与相机响应分开，具体数值应连同数据划分、测试条件和测量对象理解。The paper discusses classification, live interaction and communication experiments. Offline classification, live recognition and camera response are distinct evaluations whose numerical results depend on their data splits, conditions and measurement definitions.
我的实际工作My Contributions
- 完成手势数据采集与预处理、模型训练和嵌入式部署的研究实现。Implement the research pipeline for gesture acquisition, preprocessing, model training and embedded deployment.
- 连接 ESP32 识别输出与相机控制接口，组织交互实验。Connect ESP32 recognition output to the camera-control interface and conduct interaction experiments.
- 撰写 Camera Control System Based on TinyML Gesture Recognition，以单篇论文整理方法与实验。Author Camera Control System Based on TinyML Gesture Recognition, documenting the methods and experiments as one publication.
项目成果与验证Results & Validation
技术栈Tech Stack
工程备注 · 阶段与版本Engineering Notes · stage & revisions
- 论文、毕业设计与相关 MaixCam 实验作为同一研究工作的关联材料。TensorFlow、MaixCDK 和设备框架属于使用的工具与基础设施。The paper, thesis and related MaixCam experiments are related parts of this work. TensorFlow, MaixCDK and device frameworks are tools and infrastructure used by the project.
- 实验覆盖有限手势类别与研究条件；跨用户、姿态变化、连续动作误触发和无线丢包仍是系统局限。协议格式解析与 CRC 是不同机制。The experiments cover a limited gesture set and research conditions. Cross-user variation, orientation, unintended triggers during continuous motion and wireless loss remain limitations. Format parsing and CRC are different mechanisms.