gl-rendering

Overview ¶

SurfaceTexture GL rendering pipeline example.

Demonstrates how to use ASurfaceTexture to stream image frames from a producer (camera, video decoder) into an OpenGL ES external texture and render them on screen. This is the standard pattern for camera preview or video playback in Android NDK applications.

ASurfaceTexture bridges the Android Surface (a queue of graphic buffers) and OpenGL ES by presenting each incoming frame as a GL_TEXTURE_EXTERNAL_OES texture. The texture coordinates require a 4x4 transform matrix that compensates for buffer rotation, scaling, and coordinate-system differences.

The NDK ASurfaceTexture handle is obtained from a Java SurfaceTexture via JNI (ASurfaceTexture_fromSurfaceTexture). There is no pure-NDK constructor. Once the handle is wrapped with surfacetexture.NewSurfaceTextureFromPointer, the idiomatic API provides the full lifecycle.

GL rendering workflow:

1. Create a GL_TEXTURE_EXTERNAL_OES texture on the render thread.

2. Attach the SurfaceTexture to the current GL context with AttachToGLContext(texName). This binds the SurfaceTexture to the texture object so incoming frames update it.

3. Acquire a NativeWindow from the SurfaceTexture with AcquireWindow(). Hand this window to the frame producer (camera, codec). The producer enqueues frames into the Surface backed by this window.

4. On the render thread, when a new frame is available: a. Call UpdateTexImage() to latch the latest frame into the texture. b. Call TransformMatrix() to get the 4x4 matrix for texture coords. c. Call Timestamp() to get the frame's presentation timestamp (ns). d. Bind the texture, upload the matrix as a uniform, draw.

5. When switching GL contexts: DetachFromGLContext(), then re-attach.

6. When done: DetachFromGLContext(), then Close().

Fragment shader for GL_TEXTURE_EXTERNAL_OES:

#extension GL_OES_EGL_image_external : require
precision mediump float;
varying vec2 vTexCoord;
uniform samplerExternalOES uTexture;
uniform mat4 uTexMatrix;
void main() {
    vec2 tc = (uTexMatrix * vec4(vTexCoord, 0.0, 1.0)).xy;
    gl_FragColor = texture2D(uTexture, tc);
}

Prerequisites:

• Android device with API level 28+ (ASurfaceTexture was added in API 28).
• A Java SurfaceTexture obtained from the application's Java layer.
• A current EGL context with GL_OES_EGL_image_external support.

Because the SurfaceTexture handle requires JNI to obtain, this example documents the complete rendering pipeline and prints the API calls that a real application would make, rather than invoking them directly.

This program must run on an Android device.