I'm using CATransform3D and CAShapeLayer to create a layer like below

Here is my code.
let path = CGMutablePath()
let startPoint = CGPoint(x: center.x - width / 2, y: center.y - height / 2)
path.move(to: startPoint)
path.addLine(to: CGPoint(x: startPoint.x + width, y: startPoint.y))
path.addLine(to: CGPoint(x: startPoint.x + width, y: startPoint.y + height))
path.addLine(to: CGPoint(x: startPoint.x, y: startPoint.y + height))
path.closeSubpath()
let backgroundLayer = CAShapeLayer()
backgroundLayer.path = path
backgroundLayer.fillColor = UIColor.clear.cgColor
backgroundLayer.strokeColor = boarderColor.cgColor
var transform = CATransform3DIdentity
transform.m34 = -1 / 500
let angle = 45.toRadians
backgroundLayer.transform = CATransform3DRotate(transform, angle, 1, 0, 0)
The output is like below.
What is the reason for the difference of shape?

The backgroundLayer needs a frame and a position. If these are added, the result is as follows:
Source
Here a slightly modified version of your code that gives the result shown in the screenshot.