Slick codegen & tables with > 22 columns

Question

I'm new to Slick. I'm creating a test suite for a Java application with Scala, ScalaTest and Slick. I'm using slick to prepare data before the test and to do assertions on the data after the test. The database used has some tables with more than 22 columns. I use slick-codegen to generate my schema code.

For tables with more than 22 columns, slick-codegen does not generate a case class, but a HList-based custom type and a companion ‘constructor’ method. As I understand it, this is because the limitation that tuples and case classes can only have 22 fields. The way the code is generated, the fields of a Row-object can only be accessed by index.

I have a couple of questions about this:

1. For what I understand, the 22 fields restriction for case classes is already fixed in Scala 2.11, right?
2. If that's the case, would it be possible to customize slick-codegen to generate case classes for all tables? I looked into this: I managed to set override def hlistEnabled = false in an overridden SourceCodeGenerator. But this results in Cannot generate tuple for > 22 columns, please set hlistEnable=true or override compound. So I don’t get the point of being able to disbale HList. May be the catch is in the ‘or override compound’ part, but I don't understand what that means.
3. Searching the internet on slick and 22 columns, I came across some solutions based on nested tuples. Would it be possible to customize the codegen to use this approach?
4. If generating code with case classes with > 22 fields is not a viable option, I think it would be possible to generate an ordinary class, which has an ‘accessor’ function for each column, thus providing a ‘mapping’ from index-based access to name-based access. I’d be happy to implement the generation for this myself, but I think I need some pointers where to start. I think it should be able to override the standard codegen for this. I already use an overridden SourceCodeGenerator for some custom data types. But apart from this use case, the documentation of the code generator does not help me that much.

I would really appreciate some help here. Thanks in advance!

Answer 1

I ended up further customizing slick-codegen. First, I'll answer my own questions, then I'll post my solution.

Answers to questions

1. The 22 arity limit might be lifted for case classes, it is not for tuples. And slick-codegen also generates some tuples, which I was not fully aware of when I asked the question.
2. Not relevant, see answer 1. (This might become relevant if the 22 arity limit gets lifted for tuples as well.)
3. I chose not to investigate this further, so this question remains unanswered for now.
4. This is the approach I took, eventually.

Solution: the generated code

So, I ended up generating "ordinary" classes for tables with more than 22 columns. Let me give an example of what I generate now. (Generator code follows below.) (This example has less than 22 columns, for brevity and readability reasons.)

case class BigAssTableRow(val id: Long, val name: String, val age: Option[Int] = None)

type BigAssTableRowList = HCons[Long,HCons[String,HCons[Option[Int]]], HNil]

object BigAssTableRow {
  def apply(hList: BigAssTableRowList) = new BigAssTableRow(hlist.head, hList.tail.head, hList.tail.tail.head)
  def unapply(row: BigAssTableRow) = Some(row.id :: row.name :: row.age)
}

implicit def GetResultBoekingenRow(implicit e0: GR[Long], e1: GR[String], e2: GR[Optional[Int]]) = GR{
  prs => import prs._
  BigAssTableRow.apply(<<[Long] :: <<[String] :: <<?[Int] :: HNil)
}

class BigAssTable(_tableTag: Tag) extends Table[BigAssTableRow](_tableTag, "big_ass") {
  def * = id :: name :: age :: :: HNil <> (BigAssTableRow.apply, BigAssTableRow.unapply)

  val id: Rep[Long] = column[Long]("id", O.PrimaryKey)
  val name: Rep[String] = column[String]("name", O.Length(255,varying=true))
  val age: Rep[Option[Int]] = column[Option[Int]]("age", O.Default(None))
}

lazy val BigAssTable = new TableQuery(tag => new BigAssTable(tag))

The hardest part was finding out how the * mapping works in Slick. There's not much documentation, but I found this Stackoverflow answer rather enlightening.

I created the BigAssTableRow object to make the use of HList transparent for the client code. Note that the apply function in the object overloads the apply from the case class. So I can still create entities by calling BigAssTableRow(id: 1L, name: "Foo"), while the * projection can still use the apply function that takes an HList.

So, I can now do things like this:

// I left out the driver import as well as the scala.concurrent imports 
// for the Execution context.

val collection = TableQuery[BigAssTable]
val row = BigAssTableRow(id: 1L, name: "Qwerty") // Note that I leave out the optional age

Await.result(db.run(collection += row), Duration.Inf)

Await.result(db.run(collection.filter(_.id === 1L).result), Duration.Inf)

For this code it's totally transparent wether tuples or HLists are used under the hood.

Solution: how this is generated

I'll just post my entire generator code here. It's not perfect; please let me know if you have suggestions for improvement! Huge parts are just copied from the slick.codegen.AbstractSourceCodeGenerator and related classes and then slightly changed. There are also some things that are not directly related to this question, such as the addition of the java.time.* data types and the filtering of specific tables. I left them in, because they might be of use. Also note that this example is for a Postgres database.

import slick.codegen.SourceCodeGenerator
import slick.driver.{JdbcProfile, PostgresDriver}
import slick.jdbc.meta.MTable
import slick.model.Column

import scala.concurrent.Await
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.Duration

object MySlickCodeGenerator {
  val slickDriver = "slick.driver.PostgresDriver"
  val jdbcDriver = "org.postgresql.Driver"
  val url = "jdbc:postgresql://localhost:5432/dbname"
  val outputFolder = "/path/to/project/src/test/scala"
  val pkg = "my.package"
  val user = "user"
  val password = "password"

  val driver: JdbcProfile = Class.forName(slickDriver + "$").getField("MODULE$").get(null).asInstanceOf[JdbcProfile]
  val dbFactory = driver.api.Database
  val db = dbFactory.forURL(url, driver = jdbcDriver, user = user, password = password, keepAliveConnection = true)

  // The schema is generated using Liquibase, which creates these tables that I don't want to use
  def excludedTables = Array("databasechangelog", "databasechangeloglock")

  def tableFilter(table: MTable): Boolean = {
    !excludedTables.contains(table.name.name) && schemaFilter(table.name.schema)
  }

  // There's also an 'audit' schema in the database, I don't want to use that one
  def schemaFilter(schema: Option[String]): Boolean = {
    schema match {
      case Some("public") => true
      case None => true
      case _ => false
    }
  }

  // Fetch data model
  val modelAction = PostgresDriver.defaultTables
    .map(_.filter(tableFilter))
    .flatMap(PostgresDriver.createModelBuilder(_, ignoreInvalidDefaults = false).buildModel)

  val modelFuture = db.run(modelAction)

  // customize code generator
  val codegenFuture = modelFuture.map(model => new SourceCodeGenerator(model) {

    // add custom import for added data types
    override def code = "import my.package.Java8DateTypes._" + "\n" + super.code

    override def Table = new Table(_) {
      table =>

      // Use different factory and extractor functions for tables with > 22 columns
      override def factory   = if(columns.size == 1) TableClass.elementType else if(columns.size <= 22) s"${TableClass.elementType}.tupled" else s"${EntityType.name}.apply"
      override def extractor = if(columns.size <= 22) s"${TableClass.elementType}.unapply" else s"${EntityType.name}.unapply"

      override def EntityType = new EntityTypeDef {
        override def code = {
          val args = columns.map(c =>
            c.default.map( v =>
              s"${c.name}: ${c.exposedType} = $v"
            ).getOrElse(
              s"${c.name}: ${c.exposedType}"
            )
          )
          val callArgs = columns.map(c => s"${c.name}")
          val types = columns.map(c => c.exposedType)

          if(classEnabled){
            val prns = (parents.take(1).map(" extends "+_) ++ parents.drop(1).map(" with "+_)).mkString("")
            s"""case class $name(${args.mkString(", ")})$prns"""
          } else {
            s"""
/** Constructor for $name providing default values if available in the database schema. */
case class $name(${args.map(arg => {s"val $arg"}).mkString(", ")})
type ${name}List = ${compoundType(types)}
object $name {
  def apply(hList: ${name}List): $name = new $name(${callArgs.zipWithIndex.map(pair => s"hList${tails(pair._2)}.head").mkString(", ")})
  def unapply(row: $name) = Some(${compoundValue(callArgs.map(a => s"row.$a"))})
}
          """.trim
          }
        }
      }

      override def PlainSqlMapper = new PlainSqlMapperDef {
        override def code = {
          val positional = compoundValue(columnsPositional.map(c => if (c.fakeNullable || c.model.nullable) s"<<?[${c.rawType}]" else s"<<[${c.rawType}]"))
          val dependencies = columns.map(_.exposedType).distinct.zipWithIndex.map{ case (t,i) => s"""e$i: GR[$t]"""}.mkString(", ")
          val rearranged = compoundValue(desiredColumnOrder.map(i => if(columns.size > 22) s"r($i)" else tuple(i)))
          def result(args: String) = s"$factory($args)"
          val body =
            if(autoIncLastAsOption && columns.size > 1){
              s"""
val r = $positional
import r._
${result(rearranged)} // putting AutoInc last
              """.trim
            } else {
              result(positional)
            }

              s"""
implicit def $name(implicit $dependencies): GR[${TableClass.elementType}] = GR{
  prs => import prs._
  ${indent(body)}
}
          """.trim
        }
      }

      override def TableClass = new TableClassDef {
        override def star = {
          val struct = compoundValue(columns.map(c=>if(c.fakeNullable)s"Rep.Some(${c.name})" else s"${c.name}"))
          val rhs = s"$struct <> ($factory, $extractor)"
          s"def * = $rhs"
        }
      }

      def tails(n: Int) = {
        List.fill(n)(".tail").mkString("")
      }

      // override column generator to add additional types
      override def Column = new Column(_) {
        override def rawType = {
          typeMapper(model).getOrElse(super.rawType)
        }
      }
    }
  })

  def typeMapper(column: Column): Option[String] = {
    column.tpe match {
      case "java.sql.Date" => Some("java.time.LocalDate")
      case "java.sql.Timestamp" => Some("java.time.LocalDateTime")
      case _ => None
    }
  }

  def doCodeGen() = {
    def generator = Await.result(codegenFuture, Duration.Inf)
    generator.writeToFile(slickDriver, outputFolder, pkg, "Tables", "Tables.scala")
  }

  def main(args: Array[String]) {
    doCodeGen()
    db.close()
  }
}

Answer 2

This issue is solved in Slick 3.3: https://github.com/slick/slick/pull/1889/

This solution provides def * and def ? and also supports plain SQL.

Answer 3

Update 2019-02-15: *with the release of Slick 3.3.0, as answered by @Marcus there's built-in support for code generation of tables with > 22 columns.

As of Slick 3.2.0, the simplest solution for >22 param case class is to define the default projection in the * method using mapTo instead of the <> operator (per documented unit test):

case class BigCase(id: Int,
                   p1i1: Int, p1i2: Int, p1i3: Int, p1i4: Int, p1i5: Int, p1i6: Int,
                   p2i1: Int, p2i2: Int, p2i3: Int, p2i4: Int, p2i5: Int, p2i6: Int,
                   p3i1: Int, p3i2: Int, p3i3: Int, p3i4: Int, p3i5: Int, p3i6: Int,
                   p4i1: Int, p4i2: Int, p4i3: Int, p4i4: Int, p4i5: Int, p4i6: Int)

class bigCaseTable(tag: Tag) extends Table[BigCase](tag, "t_wide") {
      def id = column[Int]("id", O.PrimaryKey)
      def p1i1 = column[Int]("p1i1")
      def p1i2 = column[Int]("p1i2")
      def p1i3 = column[Int]("p1i3")
      def p1i4 = column[Int]("p1i4")
      def p1i5 = column[Int]("p1i5")
      def p1i6 = column[Int]("p1i6")
      def p2i1 = column[Int]("p2i1")
      def p2i2 = column[Int]("p2i2")
      def p2i3 = column[Int]("p2i3")
      def p2i4 = column[Int]("p2i4")
      def p2i5 = column[Int]("p2i5")
      def p2i6 = column[Int]("p2i6")
      def p3i1 = column[Int]("p3i1")
      def p3i2 = column[Int]("p3i2")
      def p3i3 = column[Int]("p3i3")
      def p3i4 = column[Int]("p3i4")
      def p3i5 = column[Int]("p3i5")
      def p3i6 = column[Int]("p3i6")
      def p4i1 = column[Int]("p4i1")
      def p4i2 = column[Int]("p4i2")
      def p4i3 = column[Int]("p4i3")
      def p4i4 = column[Int]("p4i4")
      def p4i5 = column[Int]("p4i5")
      def p4i6 = column[Int]("p4i6")

      // HList-based wide case class mapping
      def m3 = (
        id ::
        p1i1 :: p1i2 :: p1i3 :: p1i4 :: p1i5 :: p1i6 ::
        p2i1 :: p2i2 :: p2i3 :: p2i4 :: p2i5 :: p2i6 ::
        p3i1 :: p3i2 :: p3i3 :: p3i4 :: p3i5 :: p3i6 ::
        p4i1 :: p4i2 :: p4i3 :: p4i4 :: p4i5 :: p4i6 :: HNil
      ).mapTo[BigCase]

      def * = m3
}

EDIT

So, if you then want the slick-codegen to produce huge tables using the mapTo method described above, you override the relevant parts to the code generator and add in a mapTo statement:

package your.package
import slick.codegen.SourceCodeGenerator
import slick.{model => m}


class HugeTableCodegen(model: m.Model) extends SourceCodeGenerator(model) with GeneratorHelpers[String, String, String]{


  override def Table = new Table(_) {
    table =>

    // always defines types using case classes
    override def EntityType = new EntityTypeDef{
      override def classEnabled = true
    }

    // allow compound statements using HNil, but not for when "def *()" is being defined, instead use mapTo statement
    override def compoundValue(values: Seq[String]): String = {
      // values.size>22 assumes that this must be for the "*" operator and NOT a primary/foreign key
      if(hlistEnabled && values.size > 22) values.mkString("(", " :: ", s" :: HNil).mapTo[${StringExtensions(model.name.table).toCamelCase}Row]")
      else if(hlistEnabled) values.mkString(" :: ") + " :: HNil"
      else if (values.size == 1) values.head
      else s"""(${values.mkString(", ")})"""
    }

    // should always be case classes, so no need to handle hlistEnabled here any longer
    override def compoundType(types: Seq[String]): String = {
      if (types.size == 1) types.head
      else s"""(${types.mkString(", ")})"""
    }
  }
}

You then structure the codegen code in a separate project as documented so that it generates the source at compile time. You can pass your classname as an argument to the SourceCodeGenerator you're extending:

lazy val generateSlickSchema = taskKey[Seq[File]]("Generates Schema definitions for SQL tables")
generateSlickSchema := {

  val managedSourceFolder = sourceManaged.value / "main" / "scala"
  val packagePath = "your.sql.table.package"

  (runner in Compile).value.run(
    "slick.codegen.SourceCodeGenerator", (dependencyClasspath in Compile).value.files,
    Array(
      "env.db.connectorProfile",
      "slick.db.driver",
      "slick.db.url",
      managedSourceFolder.getPath,
      packagePath,
      "slick.db.user",
      "slick.db.password",
      "true",
      "your.package.HugeTableCodegen"
    ),
    streams.value.log
  )
  Seq(managedSourceFolder / s"${packagePath.replace(".","/")}/Tables.scala")
}

Answer 4

There are few options available as you have already found out - nested tuples, conversion from Slick HList to Shapeless HList and then to case classes and so on.

I found all those options too complicated for the task and went with customised Slick Codegen to generate simple wrapper class with accessors.

Have a look at this gist.

class MyCodegenCustomisations(model: Model) extends slick.codegen.SourceCodeGenerator(model){
import ColumnDetection._


override def Table = new Table(_){
    table =>

    val columnIndexByName = columns.map(_.name).zipWithIndex.toMap
    def getColumnIndex(columnName: String): Option[Int] = {
        columnIndexByName.get(columnName)

    }

    private def getWrapperCode: Seq[String] = {
        if (columns.length <= 22) {
            //do not generate wrapper for tables which get case class generated by Slick
            Seq.empty[String]
        } else {
            val lines =
                columns.map{c =>
                    getColumnIndex(c.name) match {
                        case Some(colIndex) =>
                            //lazy val firstname: Option[String] = row.productElement(1).asInstanceOf[Option[String]]
                            val colType = c.exposedType
                            val line = s"lazy val ${c.name}: $colType = values($colIndex).asInstanceOf[$colType]"
                            line
                        case None => ""
                    }
                }
            Seq("",
                "/*",
                "case class Wrapper(private val row: Row) {",
                "// addressing HList by index is very slow, let's convert it to vector",
                "private lazy val values = row.toList.toVector",
                ""

            ) ++ lines ++ Seq("}", "*/", "")

        }
    }


    override def code: Seq[String] = {
        val originalCode = super.code
        originalCode ++ this.getWrapperCode
    }


}

}