Cloud Foundry with 3 AZ without shared storage

Question

I am trying to install Cloud Foundry on three vSphere clusters (one node each) using Bosh without having shared storage. The nodes use their own local storages.

If I use scale-to-one-az.yml file, I can install CF and it works. If I deploy to three AZ, I am getting an error:

Error: CPI error 'Bosh::Clouds::CloudError' with message 'No valid placement found for VM compute and storage requirement' in 'create_vm' CPI method (CPI request ID: 'cpi-717770')

Can I deploy Cloud Foundry with 3 AZ without shared storage?

Additional information:

1. Cloud Foundry manifest_version: v13.23.0
2. Bosh version: version 6.3.1-c44e8a1d-2020-07-09T21:08:12Z
3. bosh cloud-config output:

azs:
- cloud_properties:
    datacenters:
    - clusters:
      - AZ01:
          datastore_pattern: node1-datastore
          persistent_datastore_pattern: node1-datastore
          resource_pool: Pool01
      name: DC
  name: z1
- cloud_properties:
    datacenters:
    - clusters:
      - AZ02:
          datastore_pattern: node2-datastore
          persistent_datastore_pattern: node2-datastore
          resource_pool: Pool02
      name: DC
  name: z2
- cloud_properties:
    datacenters:
    - clusters:
      - AZ03:
          datastore_pattern: node3-datastore
          persistent_datastore_pattern: node3-datastore
          resource_pool: Pool03
      name: DC
  name: z3
compilation:
  az: z1
  network: default
  reuse_compilation_vms: true
  vm_type: small-highmem
  workers: 6
disk_types:
- disk_size: 5120
  name: 5GB
- disk_size: 10240
  name: 10GB
- disk_size: 102400
  name: 100GB
- disk_size: 1024000
  name: 1000GB
networks:
- name: default
  subnets:
  - az: z1
    cloud_properties:
      name: LS-Cloud01
    dns:
    - 8.8.8.8
    gateway: x.x.10.1
    range: x.x.10.0/24
    reserved:
    - x.x.10.1-x.x.10.49
    static:
    - x.x.10.50 - x.x.10.60
  - az: z2
    cloud_properties:
      name: LS-Cloud02
    dns:
    - 8.8.8.8
    gateway: x.x.40.1
    range: x.x.40.0/24
    reserved:
    - x.x.40.1 - x.x.40.49
    static:
    - x.x.40.50 - x.x.40.60
  - az: z3
    cloud_properties:
      name: LS-Cloud03
    dns:
    - 8.8.8.8
    gateway: x.x.50.1
    range: x.x.50.0/24
    reserved:
    - x.x.50.1 - x.x.50.49
    static:
    - x.x.50.50 - x.x.50.60
vm_extensions:
- name: cf-router-network-properties
- name: cf-tcp-router-network-properties
- name: diego-ssh-proxy-network-properties
- cloud_properties:
    disk: 51200
  name: 50GB_ephemeral_disk
- cloud_properties:
    disk: 102400
  name: 100GB_ephemeral_disk
vm_types:
- cloud_properties:
    cpu: 2
    disk: 10240
    ram: 4096
  name: minimal
- cloud_properties:
    cpu: 6
    disk: 10240
    ram: 16384
  name: small
- cloud_properties:
    cpu: 6
    disk: 10240
    ram: 65536
  name: small-highmem

Succeeded

Answer 1

yes, it can be done.

Storage configuration (node1-datastore, node2-datastore, node3-datastore) and information about the resource pools (Pool01, Pool02, Pool03) and AZ (AZ01, AZ02, AZ03)need to be provided to bosh. My cpi.yml (https://github.com/cloudfoundry/bosh-deployment/blob/master/vsphere/cpi.yml) can be found below:

- name: cpi
  path: /releases/-
  type: replace
  value:
    name: bosh-vsphere-cpi
    sha1: 25c53531bf9efeb86d093c2acded62f638e12f0f
    url: https://bosh.io/d/github.com/cloudfoundry/bosh-vsphere-cpi-release?v=54.1.0
    version: 54.1.0
- name: stemcell
  path: /resource_pools/name=vms/stemcell?
  type: replace
  value:
    sha1: 025432cad0600ee0c05d185bc64b88d250c65de4
    url: https://bosh-core-stemcells.s3-accelerate.amazonaws.com/621.85/bosh-stemcell-621.85-vsphere-esxi-ubuntu-xenial-go_agent.tgz
- path: /resource_pools/name=vms/cloud_properties?
  type: replace
  value:
    cpu: 2
    disk: 40000
    ram: 4096
- path: /networks/name=default/subnets/0/cloud_properties?
  type: replace
  value:
    name: ((network_name))
- path: /instance_groups/name=bosh/jobs/-
  type: replace
  value:
    name: vsphere_cpi
    release: bosh-vsphere-cpi
- path: /instance_groups/name=bosh/properties/director/cpi_job?
  type: replace
  value: vsphere_cpi
- path: /cloud_provider/template?
  type: replace
  value:
    name: vsphere_cpi
    release: bosh-vsphere-cpi
- path: /instance_groups/name=bosh/properties/vcenter?
  type: replace
  value:
    address: ((vcenter_ip))
    datacenters:
    - clusters:
      - ((vcenter_cluster)): {}
      - AZ01:
          resource_pool: Pool01
      - AZ02:
          resource_pool: Pool02
      - AZ03:
          resource_pool: Pool03
      datastore_pattern: ((vcenter_ds))
      disk_path: ((vcenter_disks))
      name: ((vcenter_dc))
      persistent_datastore_pattern: ((vcenter_ds))
      template_folder: ((vcenter_templates))
      vm_folder: ((vcenter_vms))
    password: ((vcenter_password))
    user: ((vcenter_user))
- path: /cloud_provider/properties/vcenter?
  type: replace
  value:
    address: ((vcenter_ip))
    datacenters:
    - clusters:
      - ((vcenter_cluster)): {}
      - AZ01:
          resource_pool: Pool01
      - AZ02:
          resource_pool: Pool02
      - AZ03:
          resource_pool: Pool03
      datastore_pattern: ((vcenter_ds))
      disk_path: ((vcenter_disks))
      name: ((vcenter_dc))
      persistent_datastore_pattern: ((vcenter_ds))
      template_folder: ((vcenter_templates))
      vm_folder: ((vcenter_vms))
    password: ((vcenter_password))
    user: ((vcenter_user))

where vcenter_ds: node1-datastore|node2-datastore|node3-datastore

bosh cloud-config looks following:

azs:
- cloud_properties:
    datacenters:
    - clusters:
      - AZ01:
          resource_pool: Pool01
      name: DC
  name: z1
- cloud_properties:
    datacenters:
    - clusters:
      - AZ02:
          resource_pool: Pool02
      name: DC
  name: z2
- cloud_properties:
    datacenters:
    - clusters:
      - AZ03:
          resource_pool: Pool03
      name: DC
  name: z3