ECE 576 Final Project: Laser Tracker

Our project consists of using a CCD camera, Altera DE2 board, and a VGA monitor. The CCD camera captures images and sends them to the Cyclone II FPGA. Verilog code on the FPGA converts the raw image to RGB and then stores it into SDRAM. After the image has been stored, the FPGA reads the image from SDRAM. We then determine the intensity of specific pixels to track light sources and we determine what color and where we want the pixel to be displayed on the VGA. The tracking box will initially be green and when a light source is locked on, the tracking box will turn blue. The image data is then sent to an SRAM VGA buffer and then it is displayed onto the VGA during a horizontal or vertical sync. A high level block diagram of the hardware is shown below.

The camera we used was a CCD camera that uses a CMOS sensor to produce raw data. The raw data to RGB. During the RGB conversion process, each pixel's red, green, and blue components are determined. Once these three components are determined, the image is then mirrored to correct the image display and stored into SDRAM. The image data is then read out of SDRAM sequentially in a FIFO manner and placed into SRAM VGA buffer. Its coordinates on the screen corresponds directly with its physical address in SDRAM. Once there is either a horizontal sync or vertical sync, the pixels are blasted onto the VGA screen. A flow diagram of how the camera captures, processes, and displays and image is shown below in figure 1.

The image is converted from color to black and white when the image is being mirrored. When the image is being mirrored initially, each of the red, green, and blue components are being flipped and then stored into SDRAM. Instead, we only take the bottom 10 bits of the red component and discard the green and blue components. When displaying the image, these 10 bits are used for all three VGA color components: red, green, and blue. Making all three components the same makes the image black and white.

Since we need to overlay the tracking box on top of the video display to show how it tracks moving light sources, we needed a way to decide whether, given a location on-screen, it should display a red pixel, a green pixel, a blue pixel or BW pixel representing data coming from the camera image stored in SDRAM. Since the SRAM data bus is 16 bits long and the black and white image only requires the 10 most significant bits, we concatenate it with 6 extra bits, which carry out different functions depending on how they are set (see Table 1).

Table 1:

// Show SRAM on the VGA
CCD_MCLK <= ~CCD_MCLK; //25 Mhz

if (SRAM_DQ[5]==1)
begin
inRed <= 10'hFFFF;
inGreen <= 10'h0000;
inBlue <= 10'h0000;
end

else if (SRAM_DQ[4]==1)
begin
inRed <= 10'h0000;
inGreen <= 10'hFFFF;
inBlue <= 10'h0000;
end

else if (SRAM_DQ[3]==1)
begin
inRed <= 10'h0000;
inGreen <= 10'h0000;
inBlue <= 10'hFFFF;
end

else
begin
inRed <= SRAM_DQ[15:6];
inGreen <= SRAM_DQ[15:6];
inBlue <= SRAM_DQ[15:6];
end
end


Since the image is in black and white, the same SRAM data is assigned to all three color channels. If the SRAM buffer is detected to have a red output, then the VGA red component is set to 10'd1023 while both of the other components are set to zero. The same is done displaying the other colors as shown in the code segment above. The VGA outputs pixels onto the screen only when there is either a horizontal or vertical sync.

To track a laser/light source, we had a box that would search for a light source and once a light source is within the box, the box will lock onto the light source and then follow that particular light source. The dimension of our tracking box was 90 x 90 pixels with an initial green border. We detect a light source by determining how many pixels inside the tracking box are near white color. A near white color is defined as a pixel having its upper 10-bit value greater than 700. If there are more than 50 and less than 500 light pixels inside the tracking box, then it is a confirmed light source and the box will become locked on and the box's border will turn blue. We instantiated an upper bound to the total number of light pixels inside the tracking box so that the tracking box will not move if the light source is so large that it covers most or the entire tracking box area.

Since the image is in black and white, the same SRAM data is assigned to all three color channels. If the SRAM buffer is detected to have a red output, then the VGA red component is set to 10'd1023 while both of the other components are set to zero. The same is done displaying the other colors as shown in the code segment above. The VGA outputs pixels onto the screen only when there is either a horizontal or vertical sync.

The VGA Controller first draws the tracking box then finishes drawing the rest of the B&W image. Once the tracking box has locked onto a light source, the border switches from green to blue. Computations are then performed to detect if the light source is moving. To determine this, we subdivided the tracking box into nine smaller boxes with dimensions 30 x 30 pixels. The sum of 'light' pixels is calculated for each of these nine smaller boxes. The sum of the number of light pixels in the nine smaller boxes is then compared with each other to determine which box has the greatest number of light pixels. So if the upper left square has the most light pixels, the tracking box will shift upwards by 34 pixels and to the left by 34 pixels, which is essentially re-centering the box around the new light source. Table 2 below shows the number of pixels the tracking box will shift according to which square has the most light pixels. Once the tracking box has updated its position, the state machine starts all over again with the drawing of the tracking box in its new position.

Table 2:

For example, if a light source was first detected in Box 2 (as in Box 2 having more white light in it compared to all the other Boxes), the whole tracking box will shift one square up so that the light source is now located back in the center of the tracking box (Box 5). Now if the light was moved so that it now appeared in Box 9, then the whole square again will shift 1 right and 1 down to again re-center the light source.

Originally, we had wanted to track a red laser/light with color display. However, we encountered many problems when displaying a color image and displaying drawings at the same time. Therefore, we decided to use black and white imaging instead. We did manage to successfully detect intense red light. To detect red light, we used vector math calculations. For a complete red pixel, the RGB coordinates are [1023, 0, 0]. To determine if a pixel is a shade of red, we calculated the distance that point is from pure red. If the distance is less than a certain threshold, then the pixel is considered a shade of red.

Since our light tracker had some difficulty tracking faster objects, we thought that storing and reading the image from SDRAM may have been too slow, causing a delay before the tracking box determines if it should reposition itself or not. Therefore, we tried speeding up the VGA control clock from 27MHz to 50MHz. That way we could make calculations independently from synchronizing the reading of the image from SDRAM. However, this attempt failed as the screen constantly flickered without displaying any distinguishable images.