AI Gesture-Controlled Robot Using Machine Learning in PictoBlox

Introduction

You’ve made mobile robots and have controlled them using your keyboard, smartphone, joystick, and whatnot. This time we’re going to level up, channel out our inner Jean Grey, and control them using the hand gestures.

Learn how to control a 2-Wheel Drive Robot with hand gestures using PictoBlox AI’s Machine Learning extension.

Ready? Set. AI!

Prerequisites to make the Gesture Controlled Robot

You’ll need the following things to make the gesture-controlled robot:

1. A 2-wheel drive robot.
2. A laptop or a computer with a camera
3. The latest version of PictoBlox
4. A good Internet connection. 

To make the 2-wheel drive robot, visit the link here to know-how.

Training the models.

To work with the Machine Learning we first need to train our models to recognize the gestures from the stage i.e. our camera feed.

1. Visit https://teachablemachine.withgoogle.com/ to train the models.
2. Click the Get Started button. A new page will open.
3. Select the Image Project tile as we will be training the model with images of our hand-gestures.
4. Let’s add the first class. You store your sample images into the classes either by using a webcam or by uploading it. In this case, we will be adding our sample images using the Webcam, thus click on the Webcam button to start the camera.
5. We’ll first make the class for the robot to move forward, thus, keep our palm open and click on Hold to Record to record the images.
6. We will make four classes:
   1. Straight: The robot should move forward if we keep our palm straight.
   2. Left: It should turn left If we tilt our hand left.
   3. Right: It should turn right If we tilt our hands right.
   4. Stop: The robot should stop if we close our fist.
7. Similarly, make the other three classes too.

With this, you’ve entered the sample image to train the data. We’ll now train the model.

Training the model

1. Click the Train Model button and wait for the model to get trained.
2. You can have a look/test the trained model.

Exporting the Trained Model

Now, that the model is perfect, let’s export the model.

1. Click on the Export Model button.
2. A popup will open. Click the Upload my model button.
3. A sharable link will appear, Copy it.

Writing the Script

Now that the models are loaded. Let’s begin by writing the script.

Setting up the Project

1. Open PictoBlox we will first add the Machine Learning Extension.
2. To work with the all-new AI and ML extension, you need to keep two things in mind:
   1. That you must sign in / login in PictoBlox from here.
   2. You must have a good internet connection.
3. Click on the board button and select evive.
4. If you using the evive board for the first time, you need to upload the firmware. Click on the link here above to know-how.
5. We have already uploaded the firmware to our robot. We now need to add the Machine Learning extension.
6. Click the Add Extension button and select the Machine Learning extension. 
7. Click the Load Model button.
8. A pop-up will open. Paste the link that you copied from the Teachable machine and click the Load Model button.

You may now see that the Machine Learning blocks are loaded.

Setting Up the Stage and evive’s screen

1. First, drag-and-drop the when flag clicked hat block from the Events palette to execute the program when the green flag is clicked.
2. As we need to recognize the hand gestures from the camera feed, place the turn () video on stage with 0 transparency block from the Machine Learning palette.
3. We will set the evive’s display to black color. Thus, place fill screen with () color from the display palette. 
4. Place a forever block to make the code run continuously.
5. From the display palette, drag and drop the set cursor at () () block to start writing from the 10,10 every time we need to write something on the screen.Hand Gesture Controlled Robot

Assigning Actions

Next, we will recognize the gestures from the stage and perform actions accordingly.

Thus, let’s make a few custom function blocks.

1. Go to My Blocks palette.
2. Click on Make a Block button.
3. First, we will make the function to make the robot move forward. Thus, write go straight into the name and click on OK.
4. You’ll see a hat block onto the scripting area. Place the run motor () in direction () with speed () % block from the actuators block.
5. Duplicate the above block and place it below it and choose 2 from the first drop-down.
6. Similarly, we will make the custom blocks to Turn Left, Turn Right, and Brake one by one.
7. Duplicate the two motor blocks and place them below Turn Left and Turn Right custom blocks.
8. Let’s make a few changes into these blocks.
9. Into the Turn Left block, select reverse from the second drop-down of the first block.
10. Into the Turn Right block, select reverse from the second drop-down of the second block.
11. Into the Brake block, drag and drop the free motor 1 block from the actuators palette. Duplicate it and choose 2 from the second block.

Completing the main script

Now, let’s move onto the main script.

1. Place an if-else block from the controls palette below the set cursor block.
2. Drag and drop the is identified class from () is () block from the machine learning palette and choose stage from the drop-down and forward from the second drop-down. This will let us recognize the forward class from the trained model.
3. Place the go straight block into the if arm. 
4. Right, Straight on the evive’s screen using the write() block from the display palette.
5. Duplicate the if block and place it into the else arm.
6. Select left from the if block and replace go straight block with the turn left block. Write Left onto evive’s display.
7. Repeat the same process for right and stop.

The script is now complete, click on the green flag to run the script.

Conclusion

With this, your AI gesture-controlled robot is set to rock ‘n’ roll. Enjoy!