Contactless gesture-based control of smart agricultural devices
DOI:
https://doi.org/10.31548/energiya1(83).2026.047Keywords:
hand gesture recognition, agricultural drone, computer vision, MediaPipe, human-machine interaction, deep learning, contactless controlAbstract
The purpose of this study is to develop a scientifically grounded approach to the design of ergonomic control systems for information models of agricultural unmanned aerial vehicles based on hand gesture recognition algorithms using modern computer vision and deep learning technologies. The proposed approach aims to improve the efficiency, accuracy, and usability of human–machine interaction in real-world agrodrone operation. The study applies system analysis methods to formalize the “user–device” interaction, as well as comparative analysis of modern software libraries and frameworks (MediaPipe, OpenCV, TensorFlow, PyTorch). Methods of deep learning, including convolutional neural networks, are used to solve hand gesture recognition tasks. Simulation modeling using Python is employed to test and validate the developed prototype. A methodology for designing an ergonomic gesture-based control system for agro-drones has been developed. It includes the formalization of the interaction model, justification of a gesture set based on ergonomic criteria, selection of software tools, and determination of neural network architecture parameters. An experimental prototype based on MediaPipe and Python has been implemented, providing real-time gesture recognition. Simulation testing using cursor control modeling was conducted to evaluate recognition accuracy and stability. The obtained results confirm the operability of the proposed approach and demonstrate its practical applicability for developing intuitive control interfaces for agro-drones.
The proposed approach improves human–machine interaction efficiency through the use of natural gesture interfaces and adaptive recognition algorithms. The developed methodology can be applied not only to agro-drones but also to other smart devices. Experimental results confirm the feasibility of applying computer vision and neural networks for contactless control systems, opening prospects for further research in improving accuracy, robustness, and integration into real cyber-physical systems.
Recieved 2025-11-20
Recieved 2026-02-01
Accepted 2026-02-11
References
1. Mujahid, A., Awan, M. J., Yasin, A., Mohammed, M. A., Damaševičius, R., Maskeliūnas, R., & Abdulkareem, K. H. (2021). Real-time hand gesture recognition based on deep learning YOLOv3 model. Applied Sciences, 11(9), 4164. https://doi.org/10.3390/app11094164
2. Iqbal, M. Z., & Campbell, A. G. (2021). From luxury to necessity: Progress of touchless interaction technology. Technology in Society, 67, 101796. https://doi.org/10.1016/j.techsoc.2021.101796
3. Linardakis, M., Varlamis, I., & Papadopoulos, G. T. (2025). Survey on hand gesture recognition from visual input. IEEE Access, 1. https://doi.org/10.1109/access.2025.3593428
4. Bazarevsky, V., & Zhang, F. (2019, August 19). On-device, real-time hand tracking with MediaPipe. Google Research. Retrieved from https://research.google/blog/on-device-real-time-hand-tracking-with-mediapipe/
5. Yaseen, M., Kwon, O., Kim, J., Jamil, S., Lee, J., & Ullah, F. (2024). Next-gen dynamic hand gesture recognition: MediaPipe, Inception-v3 and LSTM-based enhanced deep learning model. Electronics, 13(16), 3233. https://doi.org/10.3390/electronics13163233
6. Srinil, P., & Thongnim, P. (2024). Deep learning enhanced hand gesture recognition for efficient drone use in agriculture. International Journal of Advanced Computer Science and Applications, 15(5). https://doi.org/10.14569/ijacsa.2024.01505127
7. Hermann, J., Plückthun, M., Doğangün, A., & Hesenius, M. (2022). User-defined gesture and voice control in human-drone interaction for police operations. NordiCHI '22: Nordic Human-Computer Interaction Conference. https://doi.org/10.1145/3546155.3546661
8. Hermann, J., Dogangün, A., & Hesenius, M. (2023). Age-based differences in drone control gestures: An exploratory study. OzCHI 2023: OzCHI 2023. https://doi.org/10.1145/3638380.3638401
9. Konstantoudakis, K., et al. (2022). Drone control in AR: An intuitive system for single-handed gesture control, drone tracking, and contextualized camera feed visualization in augmented reality. Drones, 6(2), 43. https://doi.org/10.3390/drones6020043
10. Kiselov, N. (2021). Drone control via gestures using MediaPipe Hands. Google Developers Blog. Retrieved from https://developers.googleblog.com/en/drone-control-via-gestures-using-mediapipe-hands
11. Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C. L., & Grundmann, M. (2020). MediaPipe Hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214.
12. Yun, G., Kwak, H., & Kim, D. H. (2024). Single-handed gesture recognition with RGB camera for drone motion control. Applied Sciences, 14(22), 10230. https://doi.org/10.3390/app142210230
13. Lu, C., Zhang, H., Pei, Y., Xie, L., Yan, Y., Yin, E., & Jin, J. (2023). Online hand gesture detection and recognition for UAV motion planning. Machines, 11(2), 210. https://doi.org/10.3390/machines11020210
14. Nguyen, V. K., & Alba-Flores, R. (2022). UAVs control using 3D hand keypoint gestures. Proceedings of SoutheastCon 2022, 140–144. https://doi.org/10.1109/SoutheastCon48659.2022.9764030
15. Ferdousi, M. (2023, June 1). Move your mouse pointer with hand gestures. DEV Community. Retrieved from https://dev.to/orthymarjan/move-your-mouse-pointer-with-hand-gestures-1gp1
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Energy and Automation
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
All materials are disseminated under the terms of the Creative Commons Attribution 4.0 International Public License, which permits others to distribute the manuscript with proper acknowledgement of the authorship and the original publication in this journal.
