AImageReader -> Sampling swaps RGB <-> BGR, how to fix?


I use an AImageReader and a Vulkan sampler to sample the camera image in a shader, used in an AR application. I have two problems:

  1. Using the format AIMAGE_FORMAT_PRIVATE does not reliably work across all devices (because some devices apply some proprietary compressed format which just lead to garbage when being sampled).

  2. Format AIMAGE_FORMAT_YUV_420_888 is the only format that must be supported by all devices according to the documentation in NdkImage.h and indeed, this works across all devices I've tested. However, blue and red color channels are swapped on Adreno devices, i.e. RGB↔BGR and I have no idea how to establish correct sampling.

Here's my setup:

// 1) Camera stream AImageReader_newWithUsage(w, h, format /* AIMAGE_FORMAT_PRIVATE or AIMAGE_FORMAT_YUV_420_888 */, AHARDWAREBUFFER_USAGE_GPU_SAMPLED_IMAGE, maxImages, &amp;reader); // per frame: AImageReader_acquireLatestImage -&gt; AImage_getHardwareBuffer(image, &amp;ahb) // 2) Conversion + immutable sampler, from the first frame&#39;s properties VkAndroidHardwareBufferFormatPropertiesANDROID formatInfo { .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_FORMAT_PROPERTIES_ANDROID, .pNext = nullptr }; VkAndroidHardwareBufferPropertiesANDROID propertiesInfo { .sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_PROPERTIES_ANDROID, .pNext = &amp;formatInfo }; auto res = vkGetAndroidHardwareBufferPropertiesANDROID(device, buffer, &amp;propertiesInfo); if (res != VK_SUCCESS) { // error handling } VkExternalFormatANDROID externalFormat { .sType = VK_STRUCTURE_TYPE_EXTERNAL_FORMAT_ANDROID, .pNext = nullptr, .externalFormat = formatInfo.externalFormat }; VkSamplerYcbcrConversionCreateInfo convInfo { .sType = VK_STRUCTURE_TYPE_SAMPLER_YCBCR_CONVERSION_CREATE_INFO, .pNext = &amp;externalFormat, .format = formatInfo.format, // Manual YCbCr conversion in shader .ycbcrModel = VK_SAMPLER_YCBCR_MODEL_CONVERSION_YCBCR_IDENTITY, .ycbcrRange = formatInfo.suggestedYcbcrRange, .components = formatInfo.samplerYcbcrConversionComponents, .xChromaOffset = formatInfo.suggestedXChromaOffset, .yChromaOffset = formatInfo.suggestedYChromaOffset, .chromaFilter = VK_FILTER_NEAREST, .forceExplicitReconstruction = VK_FALSE }; res = vkCreateSamplerYcbcrConversion(device, &amp;convInfo, nullptr, &amp;conversion); if (res != VK_SUCCESS) { // error handling } VkSamplerYcbcrConversionInfo convSamplerInfo { .sType = VK_STRUCTURE_TYPE_SAMPLER_YCBCR_CONVERSION_INFO, .pNext = nullptr, .conversion = conversion }; VkSamplerCreateInfo samplerInfo { .sType = VK_STRUCTURE_TYPE_SAMPLER_CREATE_INFO, .pNext = &amp;convSamplerInfo, .magFilter = VK_FILTER_NEAREST, .minFilter = VK_FILTER_NEAREST, .mipmapMode = VK_SAMPLER_MIPMAP_MODE_NEAREST, .addressModeU = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE, .addressModeV = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE, .addressModeW = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE, .mipLodBias = 0.0f, .anisotropyEnable = VK_FALSE, .maxAnisotropy = 1.0f, .compareEnable = VK_FALSE, .compareOp = VK_COMPARE_OP_NEVER, .minLod = 0.0f, .maxLod = VK_LOD_CLAMP_NONE, .borderColor = VK_BORDER_COLOR_FLOAT_OPAQUE_WHITE, .unnormalizedCoordinates = VK_FALSE }; vkCreateSampler(device, &amp;samplerInfo, ...);

Sampling in shader:

// fragment shader layout(binding = 1) uniform sampler2D cam; // immutable Y&#39;CbCr sampler void main() { outColor = vec4(texture(cam, uv).rgb, 1.0); }

The validation layers do not report any issues.

What do I do wrong? How can I change my setup to use AIMAGE_FORMAT_YUV_420_888 and then reliably sample the R and B color channels in the right order?

0
Oct 9 at 10:17 PM
User Avatarj00hi
#android#android-ndk#vulkan#android-camera2#yuv

No answer found for this question yet.