This blog will be a little bit different from what I usually write about but not entirely. I am very proud of the results of what I did and I would like to share them with you. It is not a secret that I work at Wrocław University of Science and Technology. Apart from my research duties, I also run classes with my students. However, we should start the story from the beginning, that is, in the year 2019. At the dawn of 2019 we had COVID-19 outbreak. It did a lot of change in our lives and daily duties which still ought to be carried out. From the perspective of my university there were a lot of challenges. The lectures could be moved to the digital world pretty easily; however, project or laboratory classes were not that easy to move. They strongly depend on human-to-human interaction, and additionally when you work with hardware, it is even more challenging. What I did and what I will describe below was a way to respond to these difficulties. It involves a lot of hardware and not a small dose of software development.

At the beginning, I would like to refer you to an article I have written about RemoteLab. You can find it here. However, here I hope to offer a bit of a different perspective. So, let us start.

RemoteLab hardware architecture

In the figure above you can see the hardware architecture. It involves mainly two components, the first one is the Raspberry Pi to which multiple STM32-based dev boards are connected via USB 3.0 Hub. Currently, there are three Raspberry Pi RemoteLab servers, each is handling around 10 dev boards. To offer some redundancy, different development boards are spread among different servers. In case of a server failure, a given type of evaluation board is available but on different server.

Beside the dev boards, RemoteLab server also has a camera (Rasberry Pi Camera module). Thanks to it, a “live” feedback can be served along with the access to the dev boards.

RemoteLab Software Architecture

Since HW architecture was discussed, now it is time to present SW architecture. Graphical representation of it is presented in the Figure below.

 It shows four main parts of the RemoteLab system itself:

• Status server
• Video server
• Debug server
• User management service

Status server

In short, the status server could be described as a dev board health monitoring service. It was implemented using the Python Flask framework; well…, the whole RemoteLab was implemented using Python. You might ask why Python and why Flask. Python offers a lot of flexibility, it is easily maintainable, and it can be run on multiple platforms. Additionally, Python comes with a variety of different libraries such as Flask or Django for web applications. If I would develop the whole solution in, e.g., C++, then probably I would be developing it until now without deployment. Thanks to Python’s flexibility, I was able to introduce changes even during laboratory classes with students. Small changes could be done effortlessly, well almost ;). Ok, I have drifted away from the original topic. Status server… In addition to monitoring status of dev boards it provides a simple interface to e.g., reset the microcontroller, halt execution, etc. What it also provides is the path to a serial device associated with each development board. It is not that easy to establish the relationship between the development board and its serial device. How would the student know that board #12 has /dev/ttyACM5 and not /dev/ttyACM8? Moreover, the order might change if you connect different boards. To facilitate finding correct serial device belonging to given evaluation board status, the server displays this information.

Video server

As I have already mentioned, Raspberry Pi can be equipped with a dedicated camera. Thanks to it, it offers “live” video feedback. Why “”ive”, you might ask? The answer is pretty straightforward: performance, or more precisely lack of it. Raspberry Pi is a nifty little devi… computer. However, imagine that 10 students are trying to simultaneously access the live video stream. Even with low resolution, you will not be able to handle that kind of traffic without hitting RPi. Keeping this in mind, I did a small trick. Instead of shooting a video, let us take a photo! Well, to be completely honest, it will also chock the machine. In the same use case, 10 students simultaneously taking a photo, it generates a lot of traffic, the Raspberry Pi Camera Module can only take photos as fast as it can. Then, what solution was implemented instead, you might ask? The video server offers images in place of video, that is one thing. The second is buffering. Along with the photo a timestamp is saved, if the timestamp is older than the given number of seconds, a new photo is taken. This allows to reduce a number of taken images. Also, photos are not saved on the hard drive (SD card) to avoid them from deteriorating too soon.

Since the technical details were laid out, let me tell you why I have even bothered. The answer is quite simple. The whole setup is used in remote classes. Students do not have a direct way to interact with the boards, so visual feedback is something that students can relay. Let us say we want to toggle a LED. You toggle the LED, you check if the status of the LED diode changed, is it lit on, or not. Based on the video feedback, you can easily determine if what you send to microcontroller is actually executed remotely.

Debug server

I would argue that this part of the system is the most important. It starts remote debuggers for each dev board. This way you can connect to the board remotely, compile a program, and upload it to MCU. Also, debugging is also possible as if the board were lying next to you. Currently, two debuggers are supported OpenOCD and stlink. This strongly depends on the development board. I have experienced some random instability of OpenOCD. It works well with older versions of ST-Link (hardware) debugger located on the dev board; however, sometimes it just does not work well with newer ones. At first I though that it only depends on the version of the ST-Link flashed to the on-board programmer but not. I did some tests where I flashed evaluation boards with different firmwares but the results were inconclusive. What is more,maybe even this was not the case because a newer board even when listed with lsusb crashed the whole USB bus. Therefore, this might be more related to kernel drivers. What solved the issue was to use the stlink software instead. When a debug server for a board was started with it, none of such behaviour was observer.

User management service

Last but not least, user management service. A set of scripts written in Python offers user management capabilities. And most of all, it offers automation. Setting up accounts, quotas, and other things for 50 or more students at a time can be a bit of a challenge. Add to that the fact that credentials must be propagated to all servers. Having a lot of fun! And if something goes wrong, you can easily lose track.

Albeit the service is called User management service, I would include into this Server management service. The prepared tool allows to update OS, update RemoteLab toolchain, manage dev boards, etc. That functionality was achieved thanks to remote SSH sessions, it is secure and most of all reliable.

Security

A very important aspect of all things connected to the Internet is security. It can be understood on different levels, data security, privacy, safety, and more. What is important is that you might have heard that every day there are a number of security breaches, server hacks, etc. Well, it is true! Although the RemoteLab is behind the WUST (Wrocław University of Science and Technology) firewall, it does not mean that it is safe. I constantly monitor the situation and believe me or not, but daily there are several dozens of e.g., SSH brute force attacks.

To protect against it, a few measures were taken. However, I will not give details on what countermeasures were taken. Why not? For an attacker, it might be easier to find a vector of attack. I gave a few examples in the article that I have linked at the beginning of this blog post ;).

Future

Since the remote classes ceased it does not mean that there is no use for RemoteLab anymore. Students can use it to prepare for stationary classes and test their own code. In my opinion, it is a good opportunity for them to get acquainted with remote type of work and practise with hardware without purchasing it.

I hope that in the future RemoteLab will undergo an upgrade. Originally, it was written with Flask. I would like to switch to Django since it can be more manageable that way. Additionally, new features will be offered, such as serial port inside a web browser. It works pretty well, but needs some more testing before it can be released to the public. And the most limited resource is always time.