Cube Sorting Using Unity & MATLAB Computer Vision

A robotic arm simulation project that uses the Faze4 robotic arm in Unity as a virtual environment, with MATLAB handling real-time color detection via computer vision and inverse kinematics via the Robotics Toolbox.

Demo

Overview

The system works as a closed loop:

1. Unity simulates the robotic arm and a conveyor belt spawning colored cubes (red, green, blue)
2. MATLAB connects to Unity over TCP, captures a camera image from the scene, detects the cube color, computes inverse kinematics, and sends joint angles back to Unity
3. The robotic arm picks up each cube and places it in the correct sorted bin

Project Structure

cube_sorting_using_unity_matlab_cv/
└── Faze4-Robotic-arm-simulator-main/
    ├── MatlabProject/          # MATLAB scripts
    │   ├── Part3.m             # Color detection + sorting (static scene)
    │   ├── Part4.m             # Color detection + sorting (conveyor belt)
    │   ├── color_check.m       # Color detection from captured image
    │   ├── TCPInit.m           # TCP client initialization
    │   ├── writeTCP.m          # TCP write helper
    │   ├── func_data.m         # Send joint angle data to Unity
    │   ├── func_grab.m         # Send gripper open/close command
    │   ├── new_object.m        # Trigger new cube spawn in Unity
    │   └── ImageReadTCP_One.m  # Receive image frame from Unity over TCP
    └── UnityProject/           # Unity 3D project
        ├── Assets/
        │   ├── Scenes/         # Unity scenes (ImportScene, ImportScene2)
        │   ├── scripts/        # C# scripts (Server, CubeMove, RobotTest, etc.)
        │   ├── Materials/      # Cube color materials (red, green, blue)
        │   ├── Prefabs/        # Cube prefabs
        │   └── textures/       # Scene textures
        ├── Packages/
        └── ProjectSettings/

Tech Stack

Component	Technology
Simulation	Unity (C#)
Kinematics	MATLAB + Robotics Toolbox
Computer Vision	MATLAB Image Processing Toolbox
Communication	TCP/IP socket (Unity server ↔ MATLAB client)
Robot Model	Faze4 6-DOF robotic arm

Prerequisites

Unity

• Unity 2021.x or later
• No additional packages required (all dependencies in Packages/manifest.json)

MATLAB

• MATLAB R2020b or later
• Robotics Toolbox for MATLAB (RTB10) by Peter Corke
• Image Processing Toolbox

Setup & Running

1. Clone the repository

git clone https://github.com/tathagata48/cube_sorting_using_unity_matlab_cv.git
cd cube_sorting_using_unity_matlab_cv

2. Open the Unity project

1. Open Unity Hub
2. Click Add and select the Faze4-Robotic-arm-simulator-main/UnityProject folder
3. Open the project with Unity 2021.x or later

3. Install MATLAB Robotics Toolbox

Follow the RTB installation guide and ensure SerialLink, Revolute, jtraj, ikine, and transl are available.

4. Run the simulation

Step 1 — Start Unity:

• Open scene Assets/Scenes/ImportScene2 (Part 4 with conveyor) or ImportScene (Part 3)
• Press Play. Unity starts a TCP server on port 55001.

Step 2 — Run MATLAB:

cd MatlabProject
run('Part4.m')   % conveyor belt sorting
% or
run('Part3.m')   % static scene sorting

MATLAB will connect to Unity, receive images, detect cube colors, compute joint trajectories, and command the arm automatically.

How It Works

Communication Protocol

• Unity acts as a TCP server (port 55001) via Server.cs
• MATLAB acts as a TCP client, initialized with TCPInit.m
• Data flow:
  • MATLAB → Unity: joint angles (6 DOF), gripper state, spawn command
  • Unity → MATLAB: raw PNG image from the virtual camera

Color Detection (color_check.m)

The camera is positioned above the cube. MATLAB captures the image, extracts the dominant color channel, and classifies:

• 1 = Red
• 2 = Green
• 3 = Blue

Inverse Kinematics (Part3.m / Part4.m)

The Faze4 arm is modeled with 6 Revolute links using DH parameters. robot.ikine(T) computes joint angles for a target end-effector pose. Smooth trajectories are generated with jtraj.

Scenes

Scene	Description
ImportScene	Basic arm + static cubes (Parts 1–3)
ImportScene2	Arm + conveyor belt spawning cubes (Part 4)

Credits

• Robotic arm model and simulator base: Faze4 Robotic Arm Simulator by PCrnjak
• Kinematics library: Robotics Toolbox for MATLAB by Peter Corke

License

This project is licensed under the MIT License. See LICENSE for details.