Real-time 2D plotting on Windows using OpenGL

Introduction - Why on earth?

There are plenty of plotting libraries for Windows using WPF / Win Forms like Oxyplot which can be configured to work with real time data. But rather than using yet another framework, I thought it would be a great idea to do some plotting of my own while also learning something about OpenGL and modern GPUs which I knew nothing about. This post is a result of that endeavour so please bear in mind that if you just want to draw a graph in a WPF application, you don’t really have to worry about OpenGL; it can be done quite easily with a Canvas element, or GDI if you are using Win Forms.

Now that all that’s clear, lets get into the lovely details.

Example Application

The example application plots real time data received from an accelerometer that’s mounted on an embedded device. The data is streamed to the PC via a Serial COM port which happens to actually be over Bluetooth. But any COM interface (USB-CDC / RS232..) will suffice. The accelerometer chip used is from ST Microelectronics - LIS3DH

A video of the application running on my PC is here:

The project includes a Simulation Mode which allows you to test the plotting without actually using a sensor, the plot just loops through some pre-recorded data. Just select the “Simulate” CheckBox and the “Start” button to start the simulation.

Download the example code here. Which you should be able to run immediately if you have Visual Studio 2012 or above.

(Modern) OpenGL in 10 minutes

I assume that like me you don’t know anything about OpenGL. If you do, then you wouldn’t be reading this anyway right? There are plenty of good tutorials on OpenGL written by those who know a lot more than me so I shall link to some that I found useful. You only need to know the absolute basics to be able to do a 2D plot, so once you reach the point where you can draw a Triangle, you know (one point) more than required and you can return back to this post. Most of the tutorials are in C++, but don’t worry about that just concentrate on understanding the principles.

Assuming that you have glanced through one or more of the tutorials above, I’ll briefly summarize the concepts relevant to the plotting scenario.

OpenGL is basically a specification that lays out rules on how software can interact with a GPU chip (you can actually think of it as a GPU driver specification). It tries to strike a balance between being low level enough to be able to utilize all of the GPUs bells and whistles while being high level enough to be portable across different GPU chips. When GPU chip maker X releases a new chip, he/she makes sure that the drivers that they release for that chip contain APIs that are compatible with the latest OpenGL. specification. On Windows, drivers also comply with the Direct3D specification.
The OpenGL interface specifies a lot of flags which encapsulate the State or Mode of the graphics engine. Typically you change whatever flags you need, then pass the vertices and other properties of the object you want to draw to the GPU and then issue the commands to draw the objects, that will be modified according to the current state (color, alpha etc).
The GPU actually has a lot of little processors called Shaders that run at different stages of the drawing pipeline, and can run in parallel. OpenGL allows you to program these shaders via a C like language (GLSL). Generally you will create a separate shader program for each stage in the drawing pipeline. At minimum you need two shader programs - one for the Vertex shader (affects vertex properties) and another for the Fragment shader (affects pixel properties).
Shader programs are compiled and linked somewhere during the run-time of your application, usually at initialization. You can switch the programs as and when you wish too. There are two kinds of variables that you can pass from your main application to a shader program
- Attributes: These represent variables that is uniquely defined for every data point that’s input to that shader, for example every pixel in a fragment shader has a unique color/alpha value.
- Uniform variables - These variables are constant for a particular draw call. Lets say you want to scale all the vertices by a factor of 2, you would store 2 in a uniform variable.
The output of a Shader program is usually the setting of some pre-defined OpenGL variable (prefixed by gl ) or output some pre-defined value. For example the vertex shader program must minimally set the variable gl_Position for all vertices which denotes the position in space of each and every vertex that you draw. And the fragment shader must return the color for each pixel.
The fundamental data structure in OpenGL is an Array (of float[]). These arrays store everything from Vertices to colors of pixels.
Inside the application you will typically store vertices, colors etc inside Buffers/Buffer Objects. Usually you have one buffer in your main application for each attribute in the shader program. You can use an Array Object to encapsulate (bind) all the Buffer Objects related to a particular scene. So when you want to draw something you just have to bind the respective array object to the OpenGL driver and then ask it to draw the scene.
When you want to draw stuff, you simply have to transfer all of these attribute buffers to the GPU and ask it to draw the image by telling the GPU the relationship between the buffers you sent and the attributes in it’s shader programs.

Wrappers for OpenGL on Windows

There are many wrappers that allow you to access the OpenGL API on Windows via C++ or C# like GLUT, GLFW, OpenTK, SharpGL etc. Except for GLUT, all of them can be used with C#/.NET. In this example I’m using SharpGL as it has built-in support for integrating OpenGL context within a WPF application. To use SharpGL just install the package with NuGet (Install-Package SharpGL.WPF) along with GlmNet which is useful for Math stuff. SharpGL also has Visual Studio project templates that you can use to create example projects.

Even though we use SharpGL for this example, as it’s basically just a thin wrapper over OpenGL it is easy to port the code to other frameworks like OpenTK.

Application Design

These are mainly 3 components to the application - the WPF User Interface that includes all the OpenGL plotting magic, a serial communication thread that collects data from the embedded board and a small module to interpret the data received from the accelerometer into X/Y/Z “points”. We shall look at each of these in turn

User Interface and Plotting

The UI consists of two areas - one occupied by the SharpGL control where we will do the plotting and another where we will have some simple buttons to ask the device to Start/Stop communication and a ComboBox to select the COM Port we want to communicate over. Data binding can be used to fill the combobox with the available serial ports by binding the contents with the GetPortNames method in System.IO.Ports :

<ObjectDataProvider ObjectType="{x:Type ports:SerialPort}" MethodName="GetPortNames" x:Key="portNames"/>
<ComboBox  ItemsSource="{Binding Source={StaticResource portNames}}" SelectedIndex="0"  x:Name="cmbxPorts" />

Since each data point from the accelerometer has 3 axes, the display surface available to the SharpGL control should be split into 3 regions. Also as far as OpenGL is concerned the entire 2D display region ranges from the X and Y values of [-1,1] so we need to convert every acceleration point into this range (normalization) before we push the data onto the VBO.

Vertex and Fragment Shaders

The Vertex Shader takes in two 3D vectors called vPosition and vColor for every buffer object and updates the variables gl_Position and color based on the information in vPosition and vColor. color is then passed on to the Fragment Shader which is used to generate the color for every pixel.

// The Vertex Shader 
// (1) takes in a 3D point and adds 'w' to it 
// (2) takes in a 3D color variable and adds Alpha to it and sets a custom variable called 'color'
#version 330

in vec3 vPosition;
in  vec3 vColor;
out vec4 color;

void main()
{

    gl_Position =  vec4(vPosition, 1.0);
    color = vec4( vColor, 1.0);
}


// The Fragment Shader outputs takes in color and outputs color

#version 330

in vec4 color;
out vec4 outputColor;

void main()
{
    outputColor = color;
}

Plotting Data

Plotting the background grid

Since the the grid is just 4 parallel lines that divide the screen into equal portions, it’s easy to draw first. This is done in the method void CreateAndDrawGrid(OpenGL GL) First of all we need to figure out the vertices to be drawn which just consists of 8 points to give 4 lines to split the figure into 3 equal regions. The points are - [(-1,-1), (1,-1)],[(-1,-1/3), (1,-1/3) …], done in the snippet

for (int i = 0; i < (C_NUM_Y_DIV + 1) * 2; i = i + 2)
{
    xgrid[i].x = -1f; xgrid[i + 1].x = 1f;
    xgrid[i].y = C_X_MARGIN_MIN + C_Y_STEP * i / 2; xgrid[i + 1].y = C_X_MARGIN_MIN + C_Y_STEP * i / 2;
    xgrid[i].z = 0f; xgrid[i + 1].z = 0f;
}

Next we need to set the colors or each of the vertices, which is set to White. If you select two different colors then the drawn line will be a color gradient between the colors.

Once the vertices and color buffers are set up, we just need to set up an Vertex Array Object (VAO) containing the two VBOs for vertices and colors, pass this VAO to OpenGL and ask it to draw a figure between these points. That’s what the rest of the code in the method does.

////  Create the vertex array object.
vertexBufferArray = new VertexBufferArray();
vertexBufferArray.Create(GL);
vertexBufferArray.Bind(GL);

//  Create a vertex buffer for the vertex data.
var vertexDataBuffer = new VertexBuffer();
vertexDataBuffer.Create(GL);
vertexDataBuffer.Bind(GL);
vertexDataBuffer.SetData(GL, attribute_vpos, xgrid, false, 3);

//  Now do the same for the colour data.
var colourDataBuffer = new VertexBuffer();
colourDataBuffer.Create(GL);
colourDataBuffer.Bind(GL);
colourDataBuffer.SetData(GL, attribute_vcol, coldata, false, 3);

//  Bind the shader, set the matrices.
shaderProgram.Bind(GL);
shaderProgram.SetUniformMatrix4(GL, "modelview", mviewdata.to_array());

GL.DrawArrays(OpenGL.GL_LINES, 0, 8);

//  Unbind our vertex array and shader.
vertexBufferArray.Unbind(GL);
shaderProgram.Unbind(GL);

Plotting the raw data

Plotting the raw data is quite similar to what was done for the grid, only difference is that the vertices are created by mapping the accelerometer data into the [-1,1] space. The rest of the stuff - VAO / VBO is the same.

The method openGLControl_OpenGLDraw(object sender, OpenGLEventArgs args) is called automatically by the control, so every time control enters here we check if there is a new data point available to plot. If yes, that’s added to the VBO and the whole scene is redrawn. The two BOs used are: vec3[][] _data for vertices of the acceleration points and vec3[][] _dataColor that stores the colors for each vertex.

Rendering is done in the method void CreateAndPlotData(OpenGL GL, float[] raw_data, int figureID) which takes in the new data available and the subplot ID. It converts the points into the [-1,1] space and draws lines through them. Conversion is done by first normalizing the data into the range [-1/3,1/3] and then shifting the data by a fixed factor to align it into one of the subplots. i.e.

for (int i = 0; i < point.Length; i++)
{

    // 1. Normalize the point.x to range [-.33,.33] so we can fit in 3 figures in the range of [1,-1]
    _data[figNum][offset + i].y = 2 * (point[i] - Accelerometer.C_MIN_LSB) / (Accelerometer.C_MAX_LSB - Accelerometer.C_MIN_LSB) - 1; // [1,-1]

    _data[figNum][offset + i].y = _data[figNum][offset + i].y * C_Y_STEP / 2; // [-.33,.33]

    if (figNum == 0)
	_data[figNum][offset + i].y = _data[figNum][offset + i].y + C_Y_STEP + factor * C_Y_STEP;
    else if (figNum == 1)
	_data[figNum][offset + i].y += factor * C_Y_STEP;
    else if (figNum == 2)
	_data[figNum][offset + i].y = _data[figNum][offset + i].y - C_Y_STEP - factor * C_Y_STEP;
}

You can avoid having to reload some data by observing that in the BOs for the raw data, two things are fixed and can be loaded at start-up -

The colors for each of the sub-plots are fixed (to Red/Green/Blue).
The X and Z values of all the samples that will be shown on screen. The X values are limited by the total number of points we want to show on screen - C_SAMPLES_PER_FRAME

Serial Communication in Windows

The raw data comes over the COM Port and is received in the class called SerialPortAdapter. It runs a background thread that constantly tries to read bytes from the serial port using the SerialPort.Read() API. If you are running .NET 4.5 you can also read data asynchronously by accessing the SerialPort.BaseStream class, which is also shown in the example. Both methods worked fine for me.

Once one or more bytes are received, it’s pushed into a Circular Buffer and then the new bytes are passed to the Accelerometer class which can interpret the data received. The circular buffer in SerialPortAdapter discards bytes only if they have been successfully processed by Accelerometer. This is important because the Accelerometer actually needs at least 6 contiguous bytes to get a valid acceleration point.

Streaming can be started and stopped by sending a special character to the device.

Parsing Accelerometer Data

The Accelerometer class provides two important interfaces -

Provides an API to interpret the byte stream received over the serial port. Converts these bytes into acceleration points and stores them in a Queue.
Defines the Maximum and Minimum value of acceleration points possible. This allows the Drawing logic to scale the screen appropriately.

If you plan to use any other sensor, you just need to reimplement these things and everything else should scale automatically. The rest of the code could be split into another class called Sensor for a more modular design, if required.

For this example, the received stream consists of a continuous stream of acceleration points and an acceleration point contains 6 bytes, 2 bytes each per axis. The bytes are arranged in Little Endian format. So (X,Y,Z) of (0x0123,0x145,0x167) would be received as the stream: 0x23, 0x01,0x45,0x01,0x67,0x01

Final words

The code can be extended easily to use other sensors with other formats by modifying the parser and the constants in Accelerometer and some of the rendering code. You can also receive data from multiple serial ports by simply creating more instances of SerialPortAdapter and hooking them to the appropriate stream parser.

This example assumes that no bytes are lost in transmission, if a byte is lost then all the subsequent acceleration values could be misinterpreted. So to make the plotting robust to such failures, you would need to send data in frames and introduce headers, length bytes etc.

To make it convinent to use .NET colors and vec3[] arrays directly, I have added two Extension methods to the SharpGL DLL which are defined in SharpGLEx they define overloads for the OpenGL.Color() and VertexBuffer.SetData() methods.

A Circular Reference

Adventures of a vagabond electron.

Real-time 2D Plotting on Windows Using OpenGL