И еще можно делать Real-time, low-latency closed-loop feedback using markerless posture tracking
Summary: The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are (1) noninvasive, (2) low-latency, and (3) provide interfaces to trigger external hardware based on posture (i.e., not just object based-tracking). Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here, we extend the DeepLabCut pose estimation toolbox to perform closed-loop feedback based on pose estimation in real time. First, we provide a new DeepLabCut-Live! package that serves as a simple API that facilitates incorporating DeepLabCut pose estimation into new or existing software for real-time, closed-loop feedback applications. Performance of this package was extensively tested on multiple hardware configurations and achieves rates ＞ 150 FPS with reasonable image sizes (＞ 400 x 300 pixels) using GPUs. In addition to the DeepLabCut-Live! API, we provide three new software tools for implementing closed-loop feedback using DeepLabCut into experiments with ease: a stand-alone GUI (called DLC-Live! GUI), and integration of DeepLabCut into the experimental design frameworks Bonsai and AutoPilot. We benchmarked the performance of all three software packages, and observed closed loop feedback with delays as low as 10-15 ms using GPUs, and delays of 15-50 ms using inexpensive embedded systems (NVIDIA Jetson Developer Kits) or conventional computers with only CPUs. Furthermore, we describe an additional forward prediction module that can achieve sub-zero to zero-latency feedback. Altogether, these tools provide users with the ability to perform closed-loop feedback based on pose estimation with state-of-the-art performance without difficult installation procedures and with little to no additional programming required.
References
Real-time, low-latency closed-loop feedback using markerless posture tracking
Gary Kane, Gonçalo Lopes, Jonny L. Saunders, Alexander Mathis, Mackenzie W. Mathis
NEUROSCIENCE
Real-time behavioral analysis
Nature Methods volume 18, page123(2021)Cite this article
834 Accesses
9 Altmetric
DeepLabCut-Live! performs pose estimation in real time and allows closed-loop feedback on behavior.
https://www.nature.com/articles/s41592-021-01072-z
Комментарии (0)