Specify colors of ggtree labels

Question

I would like to add different colors to my tree labels. Therefore, I thought of combining the tree with a metadata-frame and it seems to work well. The only problem is that I don't get how to specifiy the colors. Currently, they are randomly chosen. If possible, I'd prefer to read in the colors as vector

library("ggplot2") 
library("ggtree")

tree <- rtree(20) 
meta <- data.frame("id" = c("t4", "t6", "t17", "t13", "t12", "t8", "t16", "t10", "t20", "t2", "t14", "t18", "t11", "t7", "t15", "t9", "t19", "t3", "t1", "t5"), 
                   "num" = c(rep("1", 5), rep("2", 4), rep("3", 6), rep("4", 5)), 
                   "col" = c(rep("orange", 5), rep("gold1", 4), rep("steelblue1", 6), rep("magenta1", 5)))


p <- ggtree(tree)

p %<+% meta +    
       geom_tiplab(aes(color=num))+   
       geom_tippoint(aes(color=num))+
       theme(legend.position = "none")

Now this looks fine for my purpose, except that the colors do not match the colors from the col vector. I tried to add the colors in ggtree(tree, aes(color=col)), but this did not work.

I tried this and directly stated the colors with a column in the meta data

p %<+% meta + 
       geom_tiplab(aes(color=col))+
       geom_tippoint(aes(color=col))+
       theme(legend.position = "none")

And I tried to add the scale_fill_manual, but again without any success

p %<+% meta +    
       geom_tiplab(aes(color=num))+   
       geom_tippoint(aes(color=num))+
       theme(legend.position = "none")+   
       scale_fill_manual(values=c("1"="orange", "2"="gold1", "3"="steelblue1", "4"="magenta1"))