Mathematical Transformation-Based Data Collection (Teleoperation Method)

In this method, a data collection process based on direct mathematical transformations was followed. A total of 15 distinct human gestures were selected and captured using a Kinect sensor as 3D skeletal data. The (x, y, z) coordinates of human joints were recorded and then transferred to a NAO V6 robot through a teleoperation framework.

The teleoperation process was employed to allow the robot to directly imitate human movements. For each recorded human action, corresponding joint angle values (in radians) were computed for the NAO robot. These joint angles were used as the target labels for model training.

Communication between the Kinect sensor and the NAO robot was established through a local MQTT server. This system enabled real-time processing of human skeletal data and its transmission to the NAO robot, making the teleoperation process possible.

The main advantage of this approach lies in its ability to model robot movements using direct teleoperation and mathematical transformations. However, the accuracy of the transformations and the impact of subtle human motion variations on the resulting model must be carefully considered.

File Naming Convention in the Dataset

For example, consider the file:
K01_A001_R01.json

• K01_: Represents the subject code. The letter "K" indicates that the file contains human skeletal data obtained via Kinect.

• A001_: Indicates the gesture code. For example, A001 refers to the "Waving Hand" gesture.

• R01: Indicates the repetition number of the recorded gesture.

Similarly, consider the file:
T01_A001_R01.json

• T01_: Represents the subject code. The letter "T" indicates that the file contains radian-based joint data obtained via teleoperation on NAO.

• A001_: Indicates the code of the imitated gesture.

• R01: Indicates the repetition number.

Note: The file K01_A001_R01.json has its label in the corresponding file T01_A001_R01.json. Differentiating between input and label files is straightforward, as the only difference lies in the initial letter of the filename.

Gesture Code and Descriptions