Practical Webgl 3D Picking in P5js: Exploit a Color Buffer

P5js has a webgl mode that allows the creation of 3D worlds. But it lacks any kind of 3D picking. So lets implement a Colour buffer 3D picking for P5js for our sketches.

move mouse over the cat cubes to pick one. The code bellow is available at gist: Webgl 3D Picking in P5js with Colour Buffer

In a 3D webgl environment when you want to click on a drawn object you have to do what is called 3D Picking. There are basically two approaches to do this:

First –– the mathematical one –– is to take the camera location, and location of the mouse on the screen and ray trace back to the world in order to see where those rays will intercept with the objects.

Second –– the quick and dirty, but not so quick, 3D color picking –– is to create an hidden canvas (in the example above the right canvas) equal to the webgl canvas of the world and duplicate the world there. WHAT? Yes, the only difference of this world is that each object was assigned with a unique color. Then, the mouse coordinates on the first canvas (left) are mapped to the second hidden canvas (right) and we retrieve the color of the pixel under those coordinates. From that unique colour we can know which object it corresponds to in the real world (left).

Implementing 3D Color picking in P5.js

1. You need to change preserveDrawingBuffer: false to preserveDrawingBuffer: true in your p5.js file manually. Right now there is no other way to set this property. This might impact the performance of your application as basically you are keeping the buffer of the canvas after the renderer has drawn it instead of discarding it. The reason you need to preserve the DrawingBuffer is that you want to go to the Webgl canvas and fetch an image of it for color matching.
2. Everything as to be duplicated, every rotation, every translation… the only things different are the fills in the second color buffer canvas. The second color buffer is constructed with a createGraphics function. It will store the 3D shapes in a canvas similar to the world canvas.
3. use resetMatrix in both canvas. Otherwise you’ll get discrepancies and won’t be able to match the objects with the colors.
4. In the example above as I have few objects (less than 256) I choose to match objects by the red channel of the color instead of the full color. Your case may vary. See the comments in the code.

Extra tips for selecting 3D objects in P5js

1. The size of the buffer canvas can be smaller than the original world. You only need to adjust the mouse position by the same amount. If your animation is running slow, try this first as the readPixels function is copying a lot of pixels information all the time. Maybe you can even avoid instantiating the pixels variable every time and reuse the old one.
2. The code in the example is not optimised at all. The applying of textures could be optimised by branching carefully the code and drawing the cubes in different orders. The tip for your code it to be always careful on the order of object creation. You might be able to get some benefits.