Hiding type parameters that are implementation details

Question

I have a class that has some type parameters that are not implementation details, and some that are.

What's the best way to deal with type parameters that should not be part of the public API without resorting to unchecked casts?

I have a class is created by a builder which picks the right helpers. In my case, a helper knows how to do bulk reads/writes between buffer-like objects.

/**
 * @param <T> a type parameter that end users of PublicClass cannot ignore.
 * @param <X> has a specific relationship to T, but
 *     is an artifact of how PublicClass is implemented.
 */
public class PublicClass<T> {
  // This constructor is called from a builder.
  // The builder just returns
  //     new PublicClass<>(aHelper, someMutableState)
  // <X> is inferred from the choice of parameters.      
  <X> PublicClass(
      Helper<T, X> helper,
      StatefulInternalObject<T, X> someMutableState) {
    ...
  }

  ...
}

So there's a public class and a public builder, and some package-private implementation detail classes.

I don't want to expose <X> to clients, but my helper and my stateful objects need to interact in ways that depend on <X>, and the helper converts types that use <T> into types that use <X>, so using unchecked casts involving <X> would make PublicClass's implementation brittle.

I can create yet another hidden class that has both sets of type parameters and then make my public API just forward to that class:

public class PublicCLass<T> {
  private final class TypedImpl<T, ?> typeSafeImplementation;

  <X> PublicClass(
      Helper<T, X> helper,
      StatefulInternalObject<T, X> someMutableState) {
    typeSafeImplementation = new TypedImpl<>(
        helper, someMutableState);
  }

  // Public API just forwards to an internal
  // implementation class.
  public SomeType0 someMethod(SomeType1 x) {
    return typeSafeImplementation(x);
  }

  ...
}

Is there any better way that doesn't require these shallow forwarding methods?