RPGGdx: Vertex painting a landscape

When I left off on the last post, I had a generator for some basic procedural geometry:

This is fine and all, but it hardly looks like an actual landscape. The big thing it’s missing is textures.

So how do we texture an infinite procedural landscape? There are a number of ways to do it, but I’ve gotten decent results with a technique called vertex painting. As I said in the last post, I’m going for a retro aesthetic, something that approximates the texture styles from the old 3D worlds of the 90’s.

Blending Between Textures

For a landscape, you’re going to need to blend between multiple textures – say, between grass and rock. There are a number of ways to do this. You can simply have a sharp, hard edge where one texture ends and the other begins:

This can work for an aesthetic based on these sharp edges, like Minecraft’s. But what if you want a more natural-looking landscape? You could create “tiles” of textures designed to blend seamlessly between one texture and another, as RPG tilesets do:

…But that’s a lot of tile combinations and effort to support just a few textures. Is there a way to blend between two textures, without fiddling around with any extra “border” textures?

The answer: Vertex painting. To understand vertex painting, we have to understand how to overlay a texture onto another using a mask.

Suppose we have a basic grass texture:

And a rock texture with an alpha channel, containing a heightmap. Brighter parts of the heightmap correspond to places on the texture where the rock juts out:

How do we blend these two textures together? We could simply overlay the rock texture on the grass texture, using the heightmap as an actual alpha channel:

…Ew. No. Right away we see that we don’t want both textures to be visible at once – it just creates an ugly, muddy mess. This is true whenever you’re using vertex painting, but especially for this particular aesthetic.

What about if we use the alpha as a one-bit mask – draw the rock texture if it’s over 0.5, and fall through to the grass texture if it’s below?

Much better. The high points on the rock appear to jut through the grass. This might be even more apparent with a better selection of textures, but these still do nicely.

There’s only one thing missing, and that’s the ability to put more or less rock in different areas. We can do this using vertex attributes. One vertex attribute – call it “rockiness”- will represent a bias towards displaying the rock texture. A “1.0” will guarantee that the rock texture is drawn; a “0.0” will guarantee that you fall through to the grass texture. a “0.5” could go either way, depending on how it’s mixed with the heightmap.

Suppose the left side of our rock texture has 0.0 rockiness, and the right side has 1.0. Then we add the rockiness to the heightmap:

This time, to make it into a one-bit mask, we draw the rock texture wherever the sum is greater than 1, and fall through to grass wherever it’s less:

Much better. It doesn’t quite have the detail a custom-made border texture might have, but it can be generated on the fly using only the grass and rock textures!

Shaders and Meshes

Up until now, I’ve been explaining this mostly in image-processing terms. But this needs to be done automatically for every polygon in the landscape, which means this “rockiness” data needs to exist inside every vertex in the model.

A vertex is just a bunch of data. The simplest kind of vertex only contains a single 3D vector representing its position:

struct vertex {
    Vector3 position;

More complex vertices might have information on texture coordinates for UV mapping, a vertex color, and the “normal” (which helps define what direction the vertex receives light from):

struct vertex {
    Vector3 position;
    Vector2 texCoord;
    Vector3 color;
    Vector3 normal;

For now, all I need is the vertex position and any components (like “rockiness”) that I want to paint onto the landscape. I’m already creating my model using a LibGDX Mesh, so I define the Mesh to use these components:

Mesh mesh = new Mesh(true, vertices.length, indices.length, new VertexAttributes(
    new VertexAttribute(VertexAttributes.Usage.Position, 3, “a_position”),
    new VertexAttribute(Shaders.USAGE_ROCKCOMPONENT, 1, “a_rockComponent”),
    new VertexAttribute(Shaders.USAGE_SNOWCOMPONENT, 1, “a_snowComponent”),
    new VertexAttribute(Shaders.USAGE_DIRTCOMPONENT, 1, “a_dirtComponent”)

There’s some LibGDX cruft in here – notably the “Usage” stuff which LibGDX uses to distinguish between different attributes. The important part is that we pack four attributes into each vertex: a 3-dimensional position, and three 1-dimensional (scalar) components, for “rockyness”, “snowymess”, and “dirtiness”, respectively. These will be used to paint rock, snow, and dirt textures on top of a base grass texture. This means that when building the model, each vertex is made up of 6 floating-point numbers. If I add a new attribute for a new texture, I’ll need to tweak the code that generates the vertices to make room for it.

I actually generate these attributes when I’m generating the heightmap. I won’t make this section even longer by going through all the code, but the gist of it is: “rockyness” is nonzero wherever there’s a steep slope, and “snowyness” is nonzero at high altitudes with low slope.

The vertex shader will receive these attributes as “a_rockComponent” (and so on), and pass them to the fragment (pixel) shader as “v_rockComponent” (and so on). The pixel shader, which computes the color of each pixel on the screen, is where the magic happens.

Using the Pixel Shader

First, for reference, here’s what our landscape looks like with a simple coloration based on height, so you can see the shape:

Let’s color this with a very simple shader, to visualize the texture components:

#ifdef GL_ES
precision mediump float;

varying float v_rockComponent;
varying float v_snowComponent;
varying float v_dirtComponent;

void main()
    gl_FragColor = vec4(v_rockComponent, v_snowComponent, v_dirtComponent, 1.0)

This simply colors the pixel red based on the rock component, green based on the snow component, and blue based on the dirt component (which is currently unused).

Then, we apply the textures themselves based on the components:

#ifdef GL_ES
precision mediump float;

varying vec2 v_texCoords;
uniform sampler2D u_grassTexture;
uniform sampler2D u_rockTexture;
uniform sampler2D u_snowTexture;
uniform sampler2D u_dirtTexture;
varying float v_rockComponent;
varying float v_snowComponent;
varying float v_dirtComponent;

varying vec3 v_position;
uniform mat4 u_worldTrans;
uniform vec4 u_cameraPosition;

uniform float u_fogDistance;
uniform vec3 u_fogColor;

void main()
    vec4 grass = texture2D(u_grassTexture, v_texCoords);
    vec4 rock = texture2D(u_rockTexture, v_texCoords);
    vec4 snow = texture2D(u_snowTexture, v_texCoords);
    vec4 dirt = texture2D(u_dirtTexture, v_texCoords);

    //TODO optimize – something faster than if statements

    gl_FragColor = grass;

    if (v_rockComponent + rock.a > 1.0) {
        gl_FragColor = rock;

    if (v_snowComponent + snow.a > 1.0) {
        gl_FragColor = snow;

    if (v_dirtComponent + dirt.a > 1.0) {
        gl_FragColor = dirt;

    float depth = gl_FragCoord.z / gl_FragCoord.w / u_fogDistance;
    gl_FragColor = mix(gl_FragColor, vec4(u_fogColor, 1.0), depth);

Not bad for a landscape that uses only 3 textures. Notice that you don’t have to transition from one texture to another immediately (in the space of a single polygon) – you can have “patches” of somewhat rocky or snowy ground if the terrain is appropriate for it. This would be more difficult to do if you were custom-making a tileset.


This is where I get into unfamiliar territory. I have a vague recollection that using if statements in a shader is a Bad Thing That You Shouldn’t Do since shaders aren’t built for flow control. I don’t know the extent of this problem. Would a mix() function (a linear interpolation) between the textures, using a boolean – so, functionally identical to my if statements – have better, worse, or about the same performance? Is this issue less of a problem on certain cards, architectures, or GLSL versions? I’m only using three if statements at the moment – how quickly would it get worse if I added more?

The bad news, and the good news, is that I can’t benchmark this effectively because I never noticed any performance hit. The game never dropped below 60 FPS due to rendering, even if I was drawing several hundred models filling the entire screen space at a fairly high resolution. Potentially, this means the performance hit from the if statements won’t be a problem in practice – or maybe my GPU happens to be good at this, and others might chug along like they’re trying to run PUBG. That’s the issue with performance: PC hardware is extremely varied.

So that’s about it for vertex painting. Next post might be about path generation (and more generally, top-down vs. bottom-up approaches to generation) or about batching thousands of tree billboards together into a single model using the vertex shader.

One comment on “RPGGdx: Vertex painting a landscape”

  • “Would a mix() function (a linear interpolation) between the textures, using a boolean – so, functionally identical to my if statements – have better, worse, or about the same performance?”

    – Yes, without testing I can say: If statements / branching cause more shader variants/materials (under the hood) and therefore more drawCalls.

    You can simply do this:

    float condition = v_rockComponent + rock.a > 1.0; // this “condition” float is 0 or 1.
    gl_FragColor *= (1 – condition); // if condition is true (1), we clear the color by multiplying with 0.
    gl_FragColor += condition * rock; // if condition is true (1) we add (+=) rock.

Leave a Reply to Kyl3r Cancel reply

Your email address will not be published. Required fields are marked *

ipv6 ready