StructInspect: Perception ML For Structures 

Note: This is a speculative article and these may not be the exact techniques that StructInspect is using for their tech stack.

If your goal is to get a job in a perception ML position, it isn’t always clear what to study to make yourself valuable. There are tons of cool technologies out there, tons of tutorials and online classes and more information than any person could consume in one lifetime. So, what do we do then?

One way that I personally use is to choose companies that I would be excited to work for and investigate their products. Consider what kind of technologies they are using to build their products and what kind of problems they might have to solve to make their products useful. From there, you can put together a few topics that might be useful to learn and, even better, build a practice project that simulates a part of their (hypothetical) tech stack.

Introducing StructInspect

Today, I’d like to introduce a company that I think is doing some really cool stuff using perception ML. StructInspect, is company in Austria working on products for structure inspection using machine learning perception techniques. I am a big fan of using computer vision specifically for smart city infrastructure and I think StructInspect is a great example of it.

StructInspect actually has a wide variety of products for tracking the state of and maintaining tunnels, bridges and other infrastructure, but the technology that I think is most exciting as someone working in perception ML is Digital Inspection. If you take a look at their Digital Inspection page, you can see that their product allows users to upload data to their platform, detect anomalies and create 3D models that they can then map the anomalies on.

The people at StructInspect have already kindly split the software into two separate components and that also works nicely for our purposes. You can think of the skills you need to learn in terms of these two:

1. AI-Assisted Damage Detection
2. 3D Modeling

Let’s discuss each one and the type of technology that you might need to learn to become an engineer that StructInspect would be excited to hire.

1. AI-Assisted Damage Detection

This name actually comes from them rather than me, but we all know that AI in this context means machine learning, and since we are working with images, most likely deep learning. As you can see in the image below, there is a border wrapping around the damage so this is likely a form of segmentation.

Segmentation is a powerful technique that creates “masks” of areas of interest in an image. A mask is kind of like a map of the original image where each pixel is labelled with a class. Take the image above: all areas that are not damaged would have their pixels labelled as 0 in the mask, where as all areas that are damaged would have their pixels labelled as 1.

To study segmentation, I would recommend first trying to prepare a dataset and train a segmentation model. A good starting point is to try training an available model like DeepLabV3 on an open source dataset such as OxfordPetsDataset (Better yet, why not try wrapping it in a Lightning Module).

Once you are comfortable with training models, try making your own dataset. You can collect some pictures of a target (humans or cars is an easy one to start with) with your smartphone and use a tool like LabelStudio to label it.

Some important areas to consider when learning segmentation are:

• What kinds of segmentation are there and what are their purposes?
• What do the labels for segmentation data look like?
• How can I visualize labels and predictions from a model to make sure they look correct?
• What are some augmentations that might make my model understand the damage better?

If you are still new to computer vision, don’t rush this part. Segmentation and deep learning for computer vision as a whole are important skills for the smart city infrastructure industry and getting comfortable with PyTorch, OpenCV, Albumentations and other related tools will take you far.

2. 3D Modeling

As far as techniques go, modeling is quite a bit more advanced than segmentation and requires a deeper understanding of not just images, but also cameras and 2D vs 3D data. There are a lot of different techniques for modeling now, but let’s look specifically at their application.

Modeling is generally a two part process. The first part is creating a point cloud of the model, which is a representation of an environment in 3D space as a group of points. The second part is creating the actual 3D model by using features from the point clouds such as detecting where a surface might be.

Keep in mind that StructInspect wants to create models of the infrastructure they inspect to scale. It is important to distinguish this from simply creating a visually appealing rendering of the infrastructure. They want users to be able to know exactly at what location damage is at and also be able to measure aspects of the damage such as how large it is. This means they need accurate representations of the infrastructure.

Point Clouds

To create an accurate representation, it is essential that we have as accurate as possible of a point cloud in the first part. Generally, there are two approaches to do that. The first is to simply use a sensor that outputs a point cloud such as a LiDAR. This is straightforward, but often more expensive than a camera. If you have access to LiDAR, absolutely spend some time learning about how it is used and how to handle the point clouds created by it.

The other approach is to generate a point cloud from images. This process is known as a structure from motion (SfM). Since it only requires a camera, it is much cheaper than using a LiDAR, although it may not always be as robust (note: this is an ongoing debate in the self-driving car world). That being said, this is a great place to start for 3D modeling techniques. Many companies use a combination of LiDAR and SfM and knowing about SfM will be extremely useful.

To get started, checkout the COLMAP library. It is constantly being updated and has many of the most cutting edge techniques for SfM. In addition to that, take some time to look into the math of how SfM works. There is a lot going on, but even having a high level overview will help you a lot in your endeavors to learn modeling.

Models

Once you have obtained a point cloud, the next step is to convert it into a 3D model. This is a highly sought after technique so many of the solutions for this are paid, but there are still a few free options. I am less familiar with this area, but for starters, you can check out the Meshroom project on github. It is also highly recommended that you look into the math behind this concept as well. Topics like PCA for 3D point clouds will come in handy.

Conclusion

If you are interested in working with a cool company like StructInspect, investigating the kind of technology it is using and digging into details such as available libraries and state of the art techniques is a great way to bolster your portfolio and show them you are interested.

Build practice projects for the technologies that you are learning and put them on your Github to demonstrate your abilities and give companies something to ask about during your interviews. Just make sure to learn the concepts behind what you put on their as well so you aren’t taken by surprise when they ask you for more details!

I hope this was a useful article and I hope to do more deep dives on companies and their technology stacks in the future.