We need sampling built-ins

[abadams]

It's annoying to write the same chunk of code over and over whenever you want to sample a Func at a floating point location. We should provide built-ins. The bit that makes this hard: these built-ins should map to using the texture sampling units when inside GPU kernels.

[kgnk]

I assumed this would be done with a different type of object, maybe named Texture/TextureParam vs. Image/ImageParam. Expr indexing could maybe be handled like the third lerp() parameter, or just as either int or float coordinate(s). It's easy to generate the CPU interpolation equivalent of the texture unit, yes?

[zvookin]

Adding a sampling primitive is a great first step. There's a fair bit of design involved, but it should be fairly straight forward to define the user visible API, an intrinsic, and then add GPU specific lowering support. (It is more complicated than lerp, but the model is basically the same.) The first part can start in pure Halide. It will be difficult to guarantee the lowering does what a hardcore shader user expects. E.g. boundary conditions for texture sampling should happen in the hardware not explicitly. One open question is whether to offer more than one quality of sampling. One could build syntactic sugar on top of this to have texture objects and perhaps a Func variant indexed by floating-point coordinates. I am reluctant to head down that path as generally texture sampling is very fast on GPUs and relatively slow on CPUs so it is sort of a lie if performance portability is a concern. (I.e. one writes the code differently for CPU and GPU because the hardware performance is so different in this area.) This is one of the reasons that shader language implementations for CPUs have had very little success. In any event, defining the sampling operations and plumbing them through the compiler is work that is largely independent of whether it is an explicit call wrapped around a Func or a new object that is indexed.

…

-Z-

[kgnk]

I think any hardware capability that texture units have should be emulated by software when targeting the CPU, with the caveat that it will be slow. It should also all be optional for that same reason. Features that are not supported on a particular GPU should be handled however Halide already handles Target::CLDouble. I'm thinking runtime assertions.

Just having the feature available for CPU, including things like boundary handling and bicubic interpolation, really simplifies code that requires it.

[kree-colemcalughlin]

I like to ping this issue just to say that this would be incredibly useful. I find myself writing a bilinear sampler too often. Here are my 2 cents on a few points:

• I've been implementing function signatures like:
  Func SampleBilinear(Func f, Func u, Func v);
  Which returns a function that samples f(u(x,y), v(x,y), c). It is nice to be able to work with sampling functions this way.
• I am looking into a bicubic sampler as well, so I think having other sampling methods would be useful.
• Could we do anything with fixed point arithmetic to get improved performance when vectorizing these calculations? So maybe u, v are int16's rather than float32's, for example.
• Can we provide any automated scheduling for samplers like this on the CPU side?

[fbriggs]

I currently have some code for bilinear interpolation. I am not getting the performance I would hope for on GPU, and I am wondering if this could be a culprit... code below:

Expr bilinear(Func src, Expr x, Expr y){
  Expr x0 = cast<int>(x);
  Expr y0 = cast<int>(y);
  Expr xR   = x - x0;
  Expr yR   = y - y0;
  Expr f00  = src(x0, y0);
  Expr f01  = src(x0, y0 + 1);
  Expr f10  = src(x0 + 1, y0);
  Expr f11  = src(x0 + 1, y0 + 1);
  Expr a1   = f00;
  Expr a2   = f10 - f00;
  Expr a3   = f01 - f00;
  Expr a4   = f00 + f11 - f10 - f01;
  return a1 + a2 * xR + a3 * yR + a4 * xR * yR;
}

Please let me know if you have any ideas about how to do this type of operation faster.

[soufianekhiat]

I currently have some code for bilinear interpolation. I am not getting the performance I would hope for on GPU, and I am wondering if this could be a culprit... code below:

Expr bilinear(Func src, Expr x, Expr y){
  Expr x0 = cast<int>(x);
  Expr y0 = cast<int>(y);
  Expr xR   = x - x0;
  Expr yR   = y - y0;
  Expr f00  = src(x0, y0);
  Expr f01  = src(x0, y0 + 1);
  Expr f10  = src(x0 + 1, y0);
  Expr f11  = src(x0 + 1, y0 + 1);
  Expr a1   = f00;
  Expr a2   = f10 - f00;
  Expr a3   = f01 - f00;
  Expr a4   = f00 + f11 - f10 - f01;
  return a1 + a2 * xR + a3 * yR + a4 * xR * yR;
}

Please let me know if you have any ideas about how to do this type of operation faster.

Does their is a way to generalize it for 1D, 2D, 3D, nD cases? And maybe add a lambda for being able to replace linear interpolation with anything else cubic, sin, ...