14 December 2023
Danielle Gatland
At Carbon Critical, we are very interested in trees and forestry for their potential to help tackle climate change by absorbing carbon dioxide from the air and by providing a sustainable resource that can help replace high emissions building materials like steel and concrete. We are helping our clients in the forestry industry to build new and better products that leverage the latest sensing technologies and developments in AI. In this blog I will explain how we have used 3D modelling to generate realistic forest images that can help reduce the time to bring new ideas and products to market.
Why data availability can be a bottleneck
One of the biggest barriers to research and development of new algorithms, software, or products is the availability of data to train and test these tools. Data collection is often a challenging, time consuming, and expensive task. This is particularly true in forestry, where collecting new data requires arranging access to forests and visiting numerous sites, often in remote locations.
Further complicating things is the effort required to obtain ‘truth’ data, such as collecting accurate measurements of trees in a forest, even if just a small representative sample plot is measured (instead of the whole forest). In some cases, obtaining accurate truth data is almost impossible. For example, definitively measuring the carbon content of a tree could require cutting it down, drying it out, burning it and measuring the quantity of heat generated. In practice, assumptions and compromises are usually made to avoid destructive testing like this.
As computer vision and deep learning experts, at Carbon Critical we are particularly interested in gathering imagery (photos and videos) of forests in a broad range of scenarios. If we could bypass the need to physically go out and collect hundreds (or thousands) of images from a variety of forests, we could focus our efforts on determining the feasibility of a proposed technical solution before spending time and money on data collection activities.
Introducing 3D modelling
Using 3D modelling, not only can we produce photorealistic images, but we can programmatically compute the truth data for a given forest or scene by analysing the objects generated in it. This is incredibly powerful because it allows us to rapidly and thoroughly explore a much broader range of scenarios than would ever be possible in the real world. In the simulated world, we have full knowledge and control over every parameter, and we can simulate anything we like by adding layers to the underlying models.
We’ve used the 3D modelling software, Blender, to generate photorealistic, high resolution images of forests, like the one shown below. With Blender, we can control the species, shape, size, and placement of every tree, the density of the forest, and the topology of the land. We can explore different planting densities, mixtures of species, and so on. We know the full 3D structure of the scene and can make precise statements about the volume of wood or tonnes of carbon in the forest. And in addition to generating photorealistic imagery, we can simulate many other sensors and processes, including LiDAR, point clouds, stereo and even multimodal / hyperspectral images.
One of the key advantages of Blender is that it supports Python scripting to update parameters of the models and rendered outputs. By combining the built-in capabilities of Blender with some custom Python scripts, we have developed a pipeline to generate random variations of forests with different sized and aged trees and planting densities. We can also (programmatically) capture simulated images of the forest from different positions and camera angles. This has allowed us to test whether machine learning can help answer specific questions, without going to the effort of manually collecting thousands of photos of different types of forests.
Read on to learn how we created forest images like this with Blender.
How good does synthetic data need to be for R&D?
Naturally, synthesised data is only useful for research and development to the extent that it adequately represents the real world. If the simulated data will be used to train machine learning algorithms for real-world deployment, then it will need to be very accurate and some form of validation may be required to ensure it is sufficiently representative for the intended use.
However, even synthetic data that isn’t super accurate can still be useful for research and development. It can be great for initial feasibility testing of a concept, for sensitivity analysis, or for answering questions like "How much better will our product work if we use a higher resolution LiDAR scanner?".
For our own product development needs, we wanted to be able to generate large forestry image datasets that we could use to train and evaluate different machine learning algorithms. We wanted to get quick feedback on our ideas and to try different types of algorithms and approaches for solving problems. In this case, it wasn’t critical for our simulations to be hyper realistic, and we anticipate that in the future we will need to at least supplement our synthetic data with some real-world data to train and validate our products for implementation.
For our initial synthetic dataset, we set ourselves the objective of generating a large set of rendered forests that are realistic enough to not be immediately obvious to a forester that they are fake.
This strategy allows us to postpone the need to gather real-world data, and to push on with what we care about most - developing cool new algorithms! We will cover that in more detail at the end of this blog. First, let's look at the process of rendering realistic images of forests.
Getting started with Blender
Getting started is always the hardest part, so if you can reuse other people’s work to get a headstart, I’d definitely recommend it! From the get-go, we knew that the most important part of our images was going to be the trees and making them look as realistic and representative as possible. So we decided to buy a set of 3D modelled trees that we could use for our forest.
For the purposes of generating photo realistic imagery of 3D models, there are large texture-mapped object and scene libraries readily available on sites like TurboSquid and Free3D. Through sites like these, we were able to find existing low-cost rendered models to use in our forest.
We decided to start by modelling a pine plantation of a single tree species with the trees planted in a grid pattern. We couldn't find an exact match for pinus radiata (the species that makes up 90% of plantation forestry in Aotearoa New Zealand), but we were able to find a 3D tree model for another pine species that provided a reasonable approximation. In the future we are hoping to model mixed-species and indigenous forests, but for now focusing on a simple forest specification seemed like a good (and easy) place to start. Single species forests are also most relevant to commercial forestry, so there is still an important use case for this modelling.
To start with I took a single 3D model of a pine tree and replicated it at 5m x 7m spacings across a 100m square grid, applying some random rotations to each tree to create some variety between them. I used a simple uniform texture for the ground and sky so they had some colour, but they weren’t fully realistic. The image below shows the ‘solid view’ of what that forest layout looks like.
And below is my first attempt at rendering an image of a forest. You can tell that it’s meant to be a forest, but there are also plenty of giveaways that it isn’t real. I won’t list them all, but for starters: the ground is too consistent, the trees are too similar to each other and don’t look like a production pine forest, and the edges of the scene where the forest ends is very obvious.
To improve the realism for my next version (see next image below), I set out to make several changes. Some of the more noticeable changes were:
I used a collection of tree models - six different tree models (from the same species) instead of just one - and I chose tree models that look more like they belong in a pine plantation.
I applied some random scaling to each individual tree (in both diameter and height) and added some variability to the exact placement of the trees (rather than being in a perfect grid).
I added some ferns to make the forest scene look more realistic.
I increased the size of the forest so the edge effects would be less visible.
Making realistic images
Below is what our virtual forest looked like after those improvements. Much better, but there was clearly still work to do. Some of the key issues we had with this model were that the bark on the trees is too light for a pine plantation, the ‘plain’ lighting and sky textures are quite noticeably not real, the edge effects are still obvious, and the images are a bit blurry.
So I went back to Blender, armed with a set of tasks to fine-tune our forest, and made improvements like:
Adjusting the colours of the trees and ferns to look more realistic.
Adding a tree thinning model to represent trees being culled from the initial grid plantation over time (more on that later!).
Adding a dynamic sky model with variable cloud cover.
Adding a lighting model to create light spots and cast shadows.
Further increasing the size of the forest to eliminate edge effects.
Increasing the resolution of our renders to produce more ‘crisp’ images.
This is what the ‘solid view’ of our forest looks like after making those changes.
And this is what our rendered forest looks like.
Wow! This is a massive improvement on the previous version of our forests.
We think it passes a ‘first glance’ test of looking like a real photo of a forest (and we’ve even tested that on our friends and family!)
At this point, we decided it was time to move on and work out how to turn this into a huge dataset of different images, representing forests of different ages and planting densities, that could be used to help drive our machine learning research.
Modelling forests with different characteristics
There are several features of real plantation forests that we wanted to represent accurately in our renders to make sure our dataset isn’t too artificial: the more realistic our training dataset is, the more certainty we can have in the results of our machine learning models.
Scaling trees as they age
A feature that was important to us was to be able to model forests of different ‘ages’. At a very basic level, that means that the size (height and diameter) of the trees in the forest would need to scale as the forest ages. We developed a model for how trees grow as they age by:
Referencing pinus radiata carbon sequestration profiles (area-weighted field measurements for NZ Emissions Trading Scheme participants) provided by the Ministry for the Environment.
Estimating an average density of trees for a pinus radiata forest of a given age.
Computing the average carbon stored per tree for a forest of a given age, by combining the above two inputs.
Computing the required scaling of the reference trees for a given aged forest, by mapping the outputs of step 3 to the assumption that, on average, a 35m tall pinus radiata tree contains approximately 5 tonnes CO2.
The following graphs show the resulting mean tree height by age profile and the average carbon density for forests of different ages.
 |  |
This height-age profile is relatively generic as it was reverse-engineered from custom forest specifications and national datasets. Actual tree growth rates vary considerably between forests based on site factors such as climate, soil, slope, aspect, wind exposure, genetics, planting density, and so on. For our purposes it was not necessary to try to capture all these specifics in our model. However it is worth noting that comprehensive growth models are available for most tree species commonly planted in Aotearoa New Zealand, so there is certainly opportunity to refine this aspect of the simulation if needed.
Tree thinning over time
Another phenomenon of forests as they age is the tree thinning process. When a plantation is established, the trees are usually planted in a grid-like fashion to achieve some predetermined starting density, for example at 3m x 4m spacings. Over the first decade after planting, a forester will typically conduct one or more thinning operations to remove the less successful trees and make space for the stronger and faster growing trees. The process of controlling tree growth is known as silviculture.
Even in the absence of human intervention through silviculture, a forest will naturally thin out as it matures: the trees compete with each other for the limited supply of light, water and nutrients, and trees that don’t get enough resources don’t survive. Other thinning might also occur due to natural events such as fire, wind or disease. Older plantations typically have about 200 stems per hectare or less, regardless of what silviculture regime was applied after the trees were first established.
We didn’t want to make specific assumptions about what silviculture regime might be applied in different forests, but we wanted to capture the general trend that young forests tend to have many more trees than older forests. So, we developed a formula to simulate the density of trees in plantations of different ages based on user inputs for an initial density (I) and a ‘30-year’ tree density (F). The chart to the right shows how that trend might play out for a few options of these user inputs.
Bringing it all together
We can bring all of this together to generate random variations of forests using Blender’s Python scripting support. For each forest that we generate, we randomise the following parameters:
Parameter | Description | Range of values |
|---|---|---|
Tree density (trees per hectare) | | |
Initial tree density | Initial density of trees when planted | 500-1500 stems per hectare |
30-year tree density | Density of trees after 30 years of silviculture activities | 250-400 stems per hectare |
Tree spacing | Spacing of initial tree planting, in an 'x' and 'y' dimension | x: up to 3m spacing y: up to 9m and no more than triple the x spacing |
Forest age | Age of the forest, which informs the mean tree height and diameter, and how much tree thinning has happened since planting | 5-50 years |
Cloud cover | Amount and shape of cloud cover, 'transparency' of clouds (eg solid white or a bit transparent) | Randomised |
Random seed | A new random number is generated for each forest, to ensure they are all totally unique | Any integer (whole number) |
And then we render several images for each forest (usually around 15 images) from different positions and camera directions. On my computer (Intel i7 laptop, 16Gb RAM, NVIDIA RTX 4050 GPU), it takes about 17 minutes to simulate a random forest and render high-resolution images from 15 different camera positions and angles (or roughly 1 minute per image). The following images show some examples for forests of different ages.
 15 years |  30 years |  45 years |
What else can we do with Blender?
A key feature of Blender that we are interested in for our project is creating depth maps of our images, to estimate the distance between the camera and the trees in the image. This can give us a better idea of the actual size of each tree. In practice, this can be done with a stereo camera and stereo vision processes: two images taken from slightly different positions can be combined to estimate the distance of each object within the image. Or by a LiDAR scanner measuring distances by analysing the time between emitting laser pulses and receiving the reflected pulses.
In Blender, we can extract a ‘true’ depth map, because Blender knows the exact position of every object in the forest. This is useful for assessing the accuracy of 3D algorithms, such as stereo, simultaneous localisation and mapping (SLAM) and structure-from-motion. Below you can see an example of a rendered image and its corresponding depth map, produced by Blender. (The lighter a pixel is in the depth image, the closer it is to the camera)
 |  |
Other applications for Blender models could be to simulate point cloud outputs from ground-based or aerial LiDAR scans, hyperspectral imaging (which divides colours into many more bands than just red, green and blue, and can include colours outside the visible range), or multimodal imaging (to capture multiple types of images at once).
How does this accelerate technical innovation?
With this work, we have been able to simulate a multitude of pine forests of varying age and stocking density that are loosely representative of plantations found in Aotearoa New Zealand. For each simulated forest we have rendered several images captured from different vantage points, to create a large and varied dataset which can support our ongoing algorithm research activities.
From an algorithm research and development perspective, the ability to generate comprehensive datasets complete with ground-truth is very valuable because it removes the data acquisition bottleneck. Creating a comparable dataset using real forests would have been almost impossible for us, given the time, money and energy required to arrange access to different forests and manually collect thousands of images from them. And best of all, we have avoided the age-old problem of how to obtain accurate ground truth information for the forest.
So instead of getting hung-up on how to gather enough data to get the project started, we can generate synthetic data as needed and get one with exploring and developing solutions. We start at the 'proof-of-concept' level, where we quickly assess and reject/refine solutions until we have one that looks promising.
If we can find a technical solution that works well for the trees in our synthetic forests, then we can expand our synthetic dataset to include a wider variety of examples, or we could go out and collect real-world data to supplement or replace our synthetic data.
A practical example
To see how this works in practice, let's suppose we are interested in developing a 3D forest sensing product that uses a stereo camera rig (i.e. two identical cameras positioned a short distance apart from each other), and we are trying to select an algorithm which best estimates the depth/distance of objects in the image.
We can use our 3D forest models to simulate 'left' and 'right' image pairs to form the inputs for testing stereo algorithms. And we can extract the 'true' depth map from our Blender model. We can then try different stereo algorithms on the same input images, compare their outputs to the ground-truth image, and calculate a measure of how accurate the answer is (e.g. based on the root-mean-square error, RMSE).
As an example, the first row of images below shows a simulated left/right stereo pair and the corresponding ground-truth image (all generated from our 3D forest model). The two rows of images below that show the result of running 6 different state-of-the-art stereo algorithms on the left / right image pair.
 Left image (from Blender) |  Right image (from Blender) |  True disparity (from Blender) |
 OpenCV Block Matching |  HitNet |  CRE Stereo |
 STereo TRansformer |  Raft Stereo |  Chang Real-time |
From a quick visual inspection, it appears that the 'CRE' and 'Raft' stereo algorithms produce the best looking results. We could check this more thoroughly by running all of the algorithms on a larger set of simulated image pairs representing a broad range of forests / conditions, quantifying the accuracy (RMSE) in each case, and computing which algorithm performed the best overall.
Using simulated data for deep learning
For other applications where simulated data is used not just for proof-of-concept trials, but also to train and deploy an algorithm in the field, some extra considerations and limitations apply. In particular, for deep learning applications, it is important to avoid 'overfitting' the trained algorithm, which could happen with fully synthetic data. It is likely necessary to gather at least some real-world data to augment the training set and/or perform validation of the trained algorithm.
In the early stages of developing a deep learning solution, simulated data can be very beneficial for driving research and development. At Carbon Critical we are using our 3D forest models and simulated datasets to help identify effective deep learning network architectures and hyper-parameters for solving technical problems of interest (such as estimating the amount of carbon in a forest from a video sequence). This is enabling us to quickly get to proof-of-concept (or not!) for our innovative ideas before we need to commit to the time and cost of gathering imagery and ground truth data for real world forests.
Next steps
We are continuing to refine and extend our 3D forest modelling capability to create more comprehensive and realistic datasets. We are especially keen to start modelling mixed-species and native forests, which are much more geometrically complex than pine plantations due to their greater diversity. We hope that the availability of rich native forest image datasets with complete ground truth data will help drive the development of new algorithms and products focused on opportunities to promote indigenous forestry, which offers the best long term outcomes for both carbon sequestration and biodiversity.
We invite anyone working in this area to get in touch. Maybe together we can accelerate the development schedule of forestry technologies!
What do you think of our forest images? Have we met our objective of generating forest images that could be mistaken for real photos?
Leave us a comment below if there’s anything obvious that gives away that our images aren’t real, or if you have any other applications for a 3D model like this.
Comments