Contents

1 An introduction to remote services and multi-agent systems

1 An introduction to remote services and multi-agent systems

In the previous session, you had an opportunity to experiment hands-on with some neural networks. You may have finished that lab session wondering how neural networks can be built into real robots, particularly low-cost robots rather than expensive cutting-edge robots found only in research labs. In this session, you will find out.

The robot simulator we’re using was originally designed to simulate the behaviour of a Lego Mindstorms EV3 controlled robot. The EV3 brick is excellent for introductory robotics, but it has limitations in terms of processor speed and memory capacity. This makes it impractical to train anything other than a small neural network on a Lego EV3 robot, although we may be able to use pre-trained models to perform ‘on-device’ classification tasks.

In general, we are often faced with the problem that we may want to run powerful programs on low-cost hardware that really isn’t up to the job. Upgrading a robot with a more powerful processor might not be a solution because it adds cost and may demand extra electrical power or create heat management issues: driving a processor hard can generate a lot of heat, and you need to remove that heat somehow. Heatsinks are heavy and take up physical space, and cooling fans are heavy, take up space, and need access to power. As you add more mass, you need more powerful motors, which themselves add mass and require more power. Bigger batteries add more mass, so you can see where this argument leads…

A possible alternative is to think about using remote services or multi-agent systems approaches. In either case, we might use a low-cost robot as a mobile agent to gather data and send that back to a more powerful computer for processing.

In the first case, we might think of the robot as a remote data collector, collecting data on our behalf and then returning that data to us in response to a request for it. In the RoboLab environment we might think of the notebook Python environment as our local computing environment and the robot as a remote service. Every so often, we might pull data from the robot by making a request for a copy of it from the notebook so that we can then analyse the data at our leisure. Alternatively, each time the robot collects a dataset, it might push a copy of the data to the notebook’s Python environment. In each case, we might think of this as ‘uploading’ data from the simulated robot back to the notebook.

In a more dynamic multi-agent case, we might consider the robot and the notebook environment to be acting as peers sending messages as and when they can between each other. For example, we might have two agents: a Lego mobile robot and a personal computer (PC), or the simulated robot and the notebook. In computational terms, agents are long-lived computational systems that can deliberate on the actions they may take in pursuit of their own goals based on their own internal state (often referred to as ‘beliefs’) and sensory inputs. Their actions are then performed by means of some sort of effector system that can act to change the state of the environment within which they reside.

In a multi-agent system, two or more agents may work together to perform some task that not only meets the (sub)goals of each individual agent, but that might also strive to attain some goal agreed on by each member of the multi-agent system. Agents may communicate by making changes to the environment, for example, by leaving a trail that other agents may follow (an effect known as stigmergy), or by passing messages between themselves directly.

To a limited extent, we may view our simulated robot / Python system as a model for a simple multi-agent system where two agents – the robot, and the classifying neural network or deliberative rule-based system, for example – work together to perform the task of classifying and identifying patterns in some environment that the individual agents could not achieve by themselves. We let the robot do what it does best – move around while logging data – and then it actively sends the data back to the notebook Python environment for processing. The Python environment processes the data using a trained neural network, or perhaps a complex rule-based system, and sends back a message to the robot giving an appropriate response. In each case, the two agents act independently, sending messages to the other party when they have data or a classification to share, rather than just responding to requests for data or a particular service as they occur.

In this session, we will explore how we might use the simulated robot as a remote data collector. Every so often, we will grab a copy of the logged data into the notebook Python environment and analyse it within the notebook.

There is quite a lot of provided code in this week’s notebooks. You are not necessarily expected to be able to create this sort of code yourself. Instead, try to focus on the process of how various tasks are broken down into smaller discrete steps, as well as how small code fragments can be combined to create ‘higher-level’ functions that perform ever more powerful tasks.

1.1 Using a pre-trained MLP to categorise light sensor array data logged from the simulator

The MNIST_Digits simulator background includes various digit images from the MNIST dataset, arranged in a grid.

Alongside each digit is a grey square, where the grey level is used to encode the actual label associated with the image. (You can see how the background was created in the Background Image Generator.ipynb notebook in the top-level backgrounds folder.)

Typically, we use the light sensor to return a single value, such as the reflected light intensity value. However, in this notebook, you will use the light sensor as a simple low-resolution camera. Rather than returning a single value, the sensor returns an array of data containing the values associated with individual pixel values from a sampled image. We can then use this square array of pixel data collected by the robot inside the simulator, rather than a single reflected-light value, as the basis for trying to detect what the robot can actually see.

Note that this low-resolution camera-like functionality is not supported by the real Lego light sensor.

Let’s start by loading in the simulator:

from nbev3devsim.load_nbev3devwidget import roboSim, eds

%load_ext nbev3devsim

In order to collect the sensor image data, if the simulated robot program print() message starts with the word image_data, then we can send light sensor array data from the left, right or both light sensors to a datalog in the notebook Python environment.

The -R (--autorun) switch in the magic at the start of the following code cell will run the program in the simulator once it has been downloaded.

%%sim_magic_preloaded -b MNIST_Digits -OA -R -x 400 -y 50

# Configure a light sensor
colorLeft = ColorSensor(INPUT_2)

# This is a command invocation rather than a print statement
print("image_data left")
# The command is responded to by
# the "Image data logged..." message display

As we’re going to be collecting data from the simulator into the notebook Python environment, we should take the precaution of clearing the notebook datalog before we start using it:

%sim_data --clear

1.1.1 Pushing the sensor array datalog from the simulator to the notebook

We can now run the data collection routine by calling a simple line magic that teleports the robot to a specific location, runs the data collection program (-R) and pushes the light sensor array data to the notebook Python environment:

%sim_magic -RA -x 400 -y 850

# Wait a moment to give data time to synchronise
import time
time.sleep(1)

We may need to wait a few moments for the program to execute and the data to be sent to the notebook Python environment.

In the current example, the simulator is pushing the light sensor array data to the notebook each time the robot sends a particular message to the simulator output window.

With the data pushed from the simulator to the notebook Python environment, we should be able to see a dataframe containing the retrieved data:

roboSim.image_data()

Each row of the dataframe represents a single captured image from one of the light sensors.

1.1.2 Previewing the sampled sensor array data (optional)

Having grabbed the data, we can explore the data as rendered images.

The data representing the image is a long list of RGB (red, green, blue) values. We can generate an image from a specific row of the dataframe, given the row index:

from nn_tools.sensor_data import generate_image, zoom_img
index = -1 # Get the last image in the dataframe

img = generate_image(roboSim.image_data(),
                     index, mode='rgb')
zoom_img(img)

If you don’t see a figure image displayed, then check that the robot is placed over a figure by reviewing the sensor array display in the simulator. If the image is there, then rerun the previous code cell to see if the data is now available. If it isn’t, then rerun the data-collecting magic cell, wait a view seconds, and then try to view the zoomed image display.

We can check the colour depth of the image by calling the .getbands() method on it:

img.getbands()

As we might expect from the robot colour sensor, this is a tri-band, RGB image.

Alternatively, we can generate an image directly as a greyscale image, either by setting the mode explicitly or by omitting it (mode=greyscale is the default setting):

img = generate_image(roboSim.image_data(), index)
zoom_img(img)

img.getbands()

The images we trained the network on were size 28 × 28 pixels. The raw images retrieved from the simulator sensor are slightly smaller, coming in at 20 × 20 pixels.

img.size

The collected image also presents a square profile around the ‘circular’ sensor view. We might thus reasonably decide that we are going to focus our attention on the 14 × 14 square area in the centre of the collected image, with top left pixel (3, 3).

zoom_img(img)

One of the advantages of using the Python PIL package is that a range of methods (that is, functions) are defined on each image object that allow us to manipulate it as an image. (We can then also access the data defining the transformed image as data if we need it in that format.)

We can preview the area in our sampled image by cropping the image to the area of interest:

img = generate_image(roboSim.image_data(), index,
                    crop=(3, 3, 17, 17))

display(img.size)
zoom_img( img )

If required, we can resize the image by passing the desired size to the generate_image() function via the resize parameter, setting it either to a specified size, such as resize=(28, 28) (that is, 28 × 28 pixels) or back to the original, uncropped image size (resize=('auto')):


img = generate_image(roboSim.image_data(), index,
                    crop=(3, 3, 17, 17),
                    resize = (28, 28))

1.1.3 Collecting multiple sample images

The MINIST_Digits simulator background contains a selection of handwritten digit images arranged in a sparse grid on the background which we shall refer to as image sampling point locations. These image locations within the background can be found at the following coordinates:

  • along rows 100 pixels apart, starting at x=100 and ending at x=2000

  • along columns 100 pixels apart, starting at y=50 and ending at y=1050.

We can collect images from this grid by using magic to teleport the robot to each sampling location and then automatically run the robot program to log the sensor data. For example, to collect images from one column of the background arrangement – that is, images with a particular x-coordinate – we need to calculate the required y-values for each sampling point.

To start, let’s just check we can generate the required y-values:

# Generate a list of integers with desired range and gap
min_value = 50
max_value = 1050
step = 100

list(range(min_value, max_value+1, step))

To help us keep track of where we are in the sample collection, we can use a visual indicator such as a progress bar.

The tqdm Python package provides a wide range of tools for displaying progress bars in Python programs. For example the tqdm.notebook.trange function enhances the range iterator with an interactive progress bar that allows us to follow the progress of the iterator:

# Provide a progress bar when iterating through the range
from tqdm.notebook import trange
import time

for i in trange(min_value, max_value, step):
    # Wait a moment
    time.sleep(0.5)

We can now create a simple script that will:

  • clear the datalog

  • iterate through the desired y locations with a visual indicator of how much progress we have made

  • use line magic to locate the robot at each step and run the already downloaded image-sampling program.

To access the value of the iterated y-value in the magic, we need to prefix it with a $ when we refer to it.

# We need to add a short delay between iterations to give
# the data time to synchronise
import time

# Clear the datalog so we know it's empty
%sim_data --clear

for _y in trange(min_value, max_value+1, step):
    %sim_magic -R -x 100 -y $_y
    # Give the data time to synchronise
    time.sleep(1)

We can view the collected samples via a pandas dataframe:

image_data_df = roboSim.image_data()
image_data_df

We can access a centrally cropped black-and-white version of an image extracted from the retrieved data by index number (--index / -i) by calling the %sim_bw_image_data magic, optionally setting the --threshold / -t value away from its default value of 127. Using the --nocrop / -n flag will prevent the autocropping behaviour.

We can convert the image to a black-and-white image by setting pixels above a specified threshold value to white (255), otherwise colouring the pixel black (0) using the generate_bw_image() function. This will select a row from the datalog at a specific location and optionally crop it to a specific area. Pixel values greater than the threshold will be set to white (255) and pixel values equal to or below the threshold will be set to 0.

from  nn_tools.sensor_data import generate_bw_image

index = -1
xx = generate_bw_image(image_data_df,
                       index,
                       threshold=100,
                       crop=(3, 3, 17, 17))

zoom_img(xx)

The %sim_bw_image_data magic performs a similar operation and can also retrieve a random image from the collected data using the --random / -r flag. (The crop limits are also assumed by the magic.)

cropped_bw_image = %sim_bw_image_data --random --threshold 100
zoom_img(cropped_bw_image)

End-user applications are simple applications created by users themselves to simplify the performance of certain tasks. Such applications may be brittle and only work in specific situations or circumstances. The code may not be as elegant, well engineered or maintainable as ‘production code’ used in applications made available to other users. One of the advantages of learning to code is the ability to create your own end-user applications.

1.1.4 Activity – Observing the effect of changing threshold value when converting the image from a greyscale to a black-and-white image (optional)

Use the following end-user application to observe the effects of setting different threshold values when creating the black-and-white binarised version of the image from the original greyscale image data.

from nn_tools.sensor_data import generate_bw_image
from ipywidgets import interact_manual

@interact_manual(threshold=(0, 255),
                 index=(0, len(image_data_df)-1))
def bw_preview(index=0, threshold=200,
               crop=False):
    # Optionally crop to the centre of the image
    _crop = (3, 3, 17, 17) if crop else None
    _original_img = generate_image(image_data_df, index)
    
    # Generate a black and white image
    _demo_img = generate_bw_image(image_data_df, index,
                                  threshold=threshold,
                                  crop=_crop)
    # %sim_bw_image_data --index -1 --threshold 100 --crop 3.3,17,17
    zoom_img( _original_img)
    zoom_img( _demo_img )

    # Preview the actual sized image
    # display(_original_img, _demo_img)

The sensor_image_focus() function is another convenience function for returning the image in the centre of the sensor array.

from nn_tools.sensor_data import sensor_image_focus

original_image = generate_image(image_data_df, index)
focal_image = sensor_image_focus( original_image )
zoom_img( focal_image )

1.2 Testing the robot sampled images using a pre-trained MLP

Having grabbed the image data, we can pre-process it as required and then present it to an appropriately trained neural network to see if the network can identify the digit it represents.

1.2.1 Loading in a previously saved MLP model

Rather than train a new model, we can load in an MLP we have trained previously. Remember that when using a neural network model, we need to make sure that we know how many inputs it expects, which in our case matches the size of presented images.

You can either use the pre-trained model that is provided in the same directory as this notebook (mlp_mnist14x14.joblib), or use your own model created in an earlier notebook.

# Load model
from joblib import load

MLP = load('mlp_mnist14x14.joblib')

Check the configuration of the MLP:

from nn_tools.network_views import network_structure
network_structure(MLP)

The 196 input features correspond to an input grid of 14 × 14 pixels.

For a square array, we get the side length as the square root of the number of features.

1.2.2 Using the pre-trained classifier to recognise sampled images

What happens if we now try to recognise images sampled from the simulator light sensor array using our previously trained MLP classifier?

import random
from nn_tools.network_views import image_class_predictor

# Get a random image index value
index = random.randint(0, len(image_data_df)-1)
        
# Generate the test image as a black-and-white image
test_image = generate_bw_image(image_data_df, index,
                               threshold=127,
                               crop=(3, 3, 17, 17))

# Display a zoomed version of the test image
zoom_img(test_image)

# Print the class prediction report
image_class_predictor(MLP, test_image);

test_image2 = %sim_bw_image_data --index -1 
# Display a zoomed version of the test image
zoom_img(test_image2)

# Print the class prediction report
image_class_predictor(MLP, test_image2);

How well did the classifier perform?

Make your own notes and observations about the MLP’s performance here. If anything strikes you as unusual, why do you think the MLP is performing the way it is?

We can create a simple interactive application to test the other images more easily:

@interact_manual(threshold=(0, 255),
                 index=(0, len(image_data_df)-1))
def test_image(index=0, threshold=200, show_image=True):
    # Create the test image
    test_image = generate_bw_image(image_data_df, index, 
                                   threshold=threshold,
                                   crop=(3, 3, 17, 17))
    
    # Generate class prediction chart
    image_class_predictor(MLP, test_image)
    
    if show_image:
        zoom_img(test_image)

In general, how well does the classifier appear to perform?

Record your own notes and observations about the behaviour and performance of the MLP here.

1.2.3 Activity – Collecting image sample data at a specific location

Write a simple line magic command to collect the image data for the handwritten digit centred on the location (600, 750).

Note that you may need to wait a short time between running the data-collection program and trying to view it.

Display a zoomed version of the image in the notebook. By observation, what digit does it represent?

Using the image_class_predictor() function, how does the trained MLP classify the image? Does this match your observation?

Increase the light sensor noise in the simulator to its maximum value and collect and test the data again. How well does the network perform this time?

Hint: data is collected into a dataframe returned by calling roboSim.image_data().

Hint: remember that you need to crop the image to a 14 × 14 array.

# Your image-sampling code here

# Your image-viewing code here

From your own observation, record which digit is represented by the image here.

# How does the trained MLP classify the image?

How well does the prediction match your observation? Is the MLP confident in its prediction?

Increase the level of light sensor noise to its maximum value and rerun the experiment:

# Collect data with noise

# Preview image with noise

# Classify image with noise

Add your own notes and observations on how well the network performed the classification task in the presence of sensor noise here.

Example discussion

Click on the arrow in the sidebar or run this cell to reveal an example discussion.

We can collect the image data by calling the %sim_magic with the -R switch so that it runs the current program directly. We also need to set the location using the -x and -y parameters.

%sim_magic -R -x 600 -y 750

The data is available in a dataframe returned by calling roboSim.image_data().

To view the result, we can zoom the display of the last-collected image in the notebook-synched datalog.

# Get data for the last image in the dataframe
index = -1 
my_img = generate_bw_image(roboSim.image_data(), index,
                        crop=(3, 3, 17, 17))
zoom_img(my_img)

By my observation, the digit represented by the image at the specified location is a figure 3.

The trained MLP classifies the object as follows:

image_class_predictor(MLP, my_img)

This appears to match my prediction.

1.3 Summary

In this notebook, you have seen how we can use the robot’s light sensor as a simple low-resolution camera to sample handwritten digit images from the background. Collecting the data from the robot, we can then convert it to an image and pre-process it before testing it with a pre-trained multi-layer perceptron.

Using captured images that are slightly offset from the centre of the image array essentially provides us with a ‘jiggled’ image, which tends to increase the classification error.

You have also seen how we can automate the way the robot collects image data by ‘teleporting’ the robot to a particular location and then sampling the data there.

In the next notebook, you will see how we can use this automation approach to collect image and class data ‘in bulk’ from the simulator.

previous

<no title>

next

2 Collecting digit image and class data from the simulator