I just deployed charmed openstack based on the latest guide from the docs, but I cannot connect the noVNC console.
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
openstack maas-controller maas-one/default 2.9.32 unsupported 10:34:25-05:00
App Version Status Scale Charm Channel Rev Exposed Message
ceph-mon 17.2.0 active 3 ceph-mon quincy/stable 109 no Unit is ready and clustered
ceph-osd 17.2.0 active 3 ceph-osd quincy/stable 534 no Unit is ready (3 OSD)
ceph-radosgw 17.2.0 active 1 ceph-radosgw quincy/stable 526 no Unit is ready
cinder 20.0.0 active 1 cinder yoga/stable 557 no Unit is ready
cinder-ceph 20.0.0 active 1 cinder-ceph yoga/stable 502 no Unit is ready
cinder-mysql-router 8.0.29 active 1 mysql-router 8.0/stable 30 no Unit is ready
dashboard-mysql-router 8.0.29 active 1 mysql-router 8.0/stable 30 no Unit is ready
glance 24.0.0 active 1 glance yoga/stable 544 no Unit is ready
glance-mysql-router 8.0.29 active 1 mysql-router 8.0/stable 30 no Unit is ready
keystone 21.0.0 active 1 keystone yoga/stable 572 no Application Ready
keystone-mysql-router 8.0.29 active 1 mysql-router 8.0/stable 30 no Unit is ready
mysql-innodb-cluster 8.0.29 active 3 mysql-innodb-cluster 8.0/stable 26 no Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
ncc-mysql-router 8.0.29 active 1 mysql-router 8.0/stable 30 no Unit is ready
neutron-api 20.1.0 active 1 neutron-api yoga/stable 526 no Unit is ready
neutron-api-mysql-router 8.0.29 active 1 mysql-router 8.0/stable 30 no Unit is ready
neutron-api-plugin-ovn 20.1.0 active 1 neutron-api-plugin-ovn yoga/stable 29 no Unit is ready
nova-cloud-controller 25.0.0 active 1 nova-cloud-controller yoga/stable 601 no Unit is ready
nova-compute 25.0.0 active 3 nova-compute yoga/stable 594 no Unit is ready
openstack-dashboard 22.1.0 active 1 openstack-dashboard yoga/stable 546 no Unit is ready
ovn-central 22.03.0 active 3 ovn-central 22.03/stable 31 no Unit is ready
ovn-chassis 22.03.0 active 3 ovn-chassis 22.03/stable 46 no Unit is ready
placement 7.0.0 active 1 placement yoga/stable 49 no Unit is ready
placement-mysql-router 8.0.29 active 1 mysql-router 8.0/stable 30 no Unit is ready
rabbitmq-server 3.9.13 active 1 rabbitmq-server 3.9/stable 149 no Unit is ready
vault 1.7.9 active 1 vault 1.7/stable 68 no Unit is ready (active: true, mlock: disabled)
vault-mysql-router 8.0.29 active 1 mysql-router 8.0/stable 30 no Unit is ready
Unit Workload Agent Machine Public address Ports Message
ceph-mon/0 active idle 0/lxd/5 10.113.1.63 Unit is ready and clustered
ceph-mon/1 active idle 1/lxd/4 10.113.1.64 Unit is ready and clustered
ceph-mon/2* active idle 2/lxd/5 10.113.1.65 Unit is ready and clustered
ceph-osd/0* active idle 0 10.113.1.32 Unit is ready (3 OSD)
ceph-osd/1 active idle 1 10.113.1.30 Unit is ready (3 OSD)
ceph-osd/2 active idle 2 10.113.1.31 Unit is ready (3 OSD)
ceph-radosgw/0* active idle 0/lxd/6 10.113.1.67 80/tcp Unit is ready
cinder/0* active idle 1/lxd/5 10.113.1.66 8776/tcp Unit is ready
cinder-ceph/0* active idle 10.113.1.66 Unit is ready
cinder-mysql-router/0* active idle 10.113.1.66 Unit is ready
glance/0* active idle 0/lxd/4 10.113.1.62 9292/tcp Unit is ready
glance-mysql-router/0* active idle 10.113.1.62 Unit is ready
keystone/0* active idle 0/lxd/3 10.113.1.57 5000/tcp Unit is ready
keystone-mysql-router/0* active idle 10.113.1.57 Unit is ready
mysql-innodb-cluster/0 active idle 0/lxd/0 10.113.1.51 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/1* active idle 1/lxd/0 10.113.1.52 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/2 active idle 2/lxd/0 10.113.1.49 Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
neutron-api/0* active idle 1/lxd/2 10.113.1.56 9696/tcp Unit is ready
neutron-api-mysql-router/0* active idle 10.113.1.56 Unit is ready
neutron-api-plugin-ovn/0* active idle 10.113.1.56 Unit is ready
nova-cloud-controller/0* active idle 1/lxd/3 10.113.1.59 8774/tcp,8775/tcp Unit is ready
ncc-mysql-router/0* active idle 10.113.1.59 Unit is ready
nova-compute/0* active idle 0 10.113.1.32 Unit is ready
ovn-chassis/2* active idle 10.113.1.32 Unit is ready
nova-compute/1 active idle 1 10.113.1.30 Unit is ready
ovn-chassis/0 active idle 10.113.1.30 Unit is ready
nova-compute/2 active idle 2 10.113.1.31 Unit is ready
ovn-chassis/1 active idle 10.113.1.31 Unit is ready
openstack-dashboard/0* active idle 2/lxd/4 10.113.1.61 80/tcp,443/tcp Unit is ready
dashboard-mysql-router/0* active idle 10.113.1.61 Unit is ready
ovn-central/0* active idle 0/lxd/2 10.113.1.54 6641/tcp,6642/tcp Unit is ready
ovn-central/1 active idle 1/lxd/1 10.113.1.53 6641/tcp,6642/tcp Unit is ready (leader: ovnnb_db, ovnsb_db)
ovn-central/2 active idle 2/lxd/1 10.113.1.55 6641/tcp,6642/tcp Unit is ready
placement/0* active idle 2/lxd/3 10.113.1.60 8778/tcp Unit is ready
placement-mysql-router/0* active idle 10.113.1.60 Unit is ready
rabbitmq-server/0* active idle 2/lxd/2 10.113.1.58 5672/tcp,15672/tcp Unit is ready
vault/0* active idle 0/lxd/1 10.113.1.50 8200/tcp Unit is ready (active: true, mlock: disabled)
vault-mysql-router/0* active idle 10.113.1.50 Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 10.113.1.32 node3 jammy default Deployed
0/lxd/0 started 10.113.1.51 juju-25c531-0-lxd-0 jammy default Container started
0/lxd/1 started 10.113.1.50 juju-25c531-0-lxd-1 jammy default Container started
0/lxd/2 started 10.113.1.54 juju-25c531-0-lxd-2 jammy default Container started
0/lxd/3 started 10.113.1.57 juju-25c531-0-lxd-3 jammy default Container started
0/lxd/4 started 10.113.1.62 juju-25c531-0-lxd-4 jammy default Container started
0/lxd/5 started 10.113.1.63 juju-25c531-0-lxd-5 jammy default Container started
0/lxd/6 started 10.113.1.67 juju-25c531-0-lxd-6 jammy default Container started
1 started 10.113.1.30 node1 jammy default Deployed
1/lxd/0 started 10.113.1.52 juju-25c531-1-lxd-0 jammy default Container started
1/lxd/1 started 10.113.1.53 juju-25c531-1-lxd-1 jammy default Container started
1/lxd/2 started 10.113.1.56 juju-25c531-1-lxd-2 jammy default Container started
1/lxd/3 started 10.113.1.59 juju-25c531-1-lxd-3 jammy default Container started
1/lxd/4 started 10.113.1.64 juju-25c531-1-lxd-4 jammy default Container started
1/lxd/5 started 10.113.1.66 juju-25c531-1-lxd-5 jammy default Container started
2 started 10.113.1.31 node2 jammy default Deployed
2/lxd/0 started 10.113.1.49 juju-25c531-2-lxd-0 jammy default Container started
2/lxd/1 started 10.113.1.55 juju-25c531-2-lxd-1 jammy default Container started
2/lxd/2 started 10.113.1.58 juju-25c531-2-lxd-2 jammy default Container started
2/lxd/3 started 10.113.1.60 juju-25c531-2-lxd-3 focal default Container started
2/lxd/4 started 10.113.1.61 juju-25c531-2-lxd-4 jammy default Container started
2/lxd/5 started 10.113.1.65 juju-25c531-2-lxd-5 jammy default Container started
From the Horizon web ui, I launched a new Ubuntu 22 instance and I am able to ssh into the machine. However, when I go to Instances>{Instance Name}>Console, noVNC seems to load, but it just says "Something went wrong, connection is closed". It does give me the option for show console the console in a new tab, but it still shows the same error. I believe noVNC is loading because it shows the "Send CtrlAltDel" button in the top right corner.
My next step was to check the services running on the nova-cloud-controller charm. This is running as an LXD container at 10.113.1.59 (see juju status log).
$ systemctl status nova-novncproxy.service
● nova-novncproxy.service - OpenStack Compute novncproxy
Loaded: loaded (/lib/systemd/system/nova-novncproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-07-12 15:32:25 UTC; 11min ago
Docs: man:nova-novncproxy(1)
Main PID: 1900710 (nova-novncproxy)
Tasks: 1 (limit: 314572)
Memory: 99.1M
CPU: 3.018s
CGroup: /system.slice/nova-novncproxy.service
└─1900710 /usr/bin/python3 /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-novncproxy.log
Jul 12 15:32:25 juju-25c531-1-lxd-3 systemd[1]: Started OpenStack Compute novncproxy.
Jul 12 15:32:25 juju-25c531-1-lxd-3 nova-novncproxy[1900710]: Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not >
Jul 12 15:32:27 juju-25c531-1-lxd-3 nova-novncproxy[1900710]: Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
$ cat /var/log/nova/nova-novncproxy.log
2022-07-12 15:32:24.759 1888563 INFO nova.console.websocketproxy [-] In exit
2022-07-12 15:32:27.790 1900710 INFO nova.console.websocketproxy [-] WebSocket server settings:
2022-07-12 15:32:27.791 1900710 INFO nova.console.websocketproxy [-] - Listen on 0.0.0.0:6080
2022-07-12 15:32:27.791 1900710 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc
2022-07-12 15:32:27.792 1900710 INFO nova.console.websocketproxy [-] - SSL/TLS support
2022-07-12 15:32:27.792 1900710 INFO nova.console.websocketproxy [-] - proxying from 0.0.0.0:6080 to None:None
2022-07-12 15:32:32.664 1900717 INFO nova.console.websocketproxy [-] 172.16.33.2 - - [12/Jul/2022 15:32:32] 172.16.33.2: SSL/TLS (wss://) WebSocket connection
2022-07-12 15:32:32.665 1900717 INFO nova.console.websocketproxy [-] 172.16.33.2 - - [12/Jul/2022 15:32:32] 172.16.33.2: Path: '/?token=21fff4d4-5596-4586-95b3-705f9116070b'
2022-07-12 15:32:32.917 1900717 INFO nova.compute.rpcapi [req-2123ff31-79c5-4d43-ab1d-0a85c81c8785 - - - - -] Automatically selected compute RPC version 6.0 from minimum service version 61
2022-07-12 15:32:32.954 1900717 INFO nova.console.websocketproxy [req-2123ff31-79c5-4d43-ab1d-0a85c81c8785 - - - - -] 2: connect info: ConsoleAuthToken(access_url_base='https://10.113.1.59:6080/vnc_auto.html',console_type='novnc',created_at=2022-07-12T15:29:38Z,host='127.0.0.1',id=21,instance_uuid=7446a69f-73e1-4787-ad11-b22be5662e44,internal_access_path=None,port=5900,token='***',updated_at=None)
2022-07-12 15:32:32.955 1900717 INFO nova.console.websocketproxy [req-2123ff31-79c5-4d43-ab1d-0a85c81c8785 - - - - -] 2: connecting to: 127.0.0.1:5900
2022-07-12 15:32:32.955 1900717 INFO nova.console.websocketproxy [req-2123ff31-79c5-4d43-ab1d-0a85c81c8785 - - - - -] handler exception: [Errno 111] ECONNREFUSED
2022-07-12 15:40:29.753 1903447 INFO nova.console.websocketproxy [-] handler exception: [SSL: UNEXPECTED_EOF_WHILE_READING] unexpected eof while reading (_ssl.c:997)
2022-07-12 15:40:30.789 1903448 INFO nova.console.websocketproxy [-] 172.16.33.2 - - [12/Jul/2022 15:40:30] 172.16.33.2: SSL/TLS (wss://) WebSocket connection
2022-07-12 15:40:30.790 1903448 INFO nova.console.websocketproxy [-] 172.16.33.2 - - [12/Jul/2022 15:40:30] 172.16.33.2: Path: '/?token=21fff4d4-5596-4586-95b3-705f9116070b'
2022-07-12 15:40:30.916 1903448 INFO nova.console.websocketproxy [req-85b9a81e-34a4-4679-ade7-f3d68e6f2f40 - - - - -] handler exception: The token '***' is invalid or has expired
I restarted the nova-novnc service, but that did not resolve the issue.
I also checked tcpdump on the compute nodes where the instance was hosted, but it did not receive any traffic on port 5900. I did the same from within the vm instance.
My best guess is that there is something configured incorrectly with the nova-cloud-controller, but I'm not sure what.
Here is the nova.conf file on the nova-cloud-controller file:
$ cat /etc/nova/nova.conf
# train
###############################################################################
# [ WARNING ]
# Configuration file maintained by Juju. Local changes may be overwritten.
###############################################################################
[DEFAULT]
verbose=False
debug=False
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
volumes_path=/var/lib/nova/volumes
enabled_apis=osapi_compute,metadata
compute_driver=libvirt.LibvirtDriver
use_ipv6 = False
osapi_compute_listen = 0.0.0.0
metadata_host = 0.0.0.0
s3_listen = 0.0.0.0
enable_new_services = True
transport_url = rabbit://nova:<*redacted*>@10.113.1.58:5672/openstack
osapi_compute_workers = 4
cpu_allocation_ratio = 2
ram_allocation_ratio = 0.98
disk_allocation_ratio = 1
use_syslog=False
my_ip = 10.113.1.59
cert=/etc/apache2/ssl/nova/cert_10.113.1.59
key=/etc/apache2/ssl/nova/key_10.113.1.59
libvirt_vif_driver = nova.virt.libvirt.vif.LibvirtGenericVIFDriver
libvirt_user_virtio_for_bridges = True
security_group_api = neutron
nova_firewall_driver = nova.virt.firewall.NoopFirewallDriver
default_floating_pool = ext_net
network_api_class = nova.network.neutronv2.api.API
volume_api_class=nova.volume.cinder.API
osapi_compute_listen_port = 8754
metadata_listen_port = 8755
[upgrade_levels]
compute = auto
[database]
connection = mysql+pymysql://nova:<*redacted*>@127.0.0.1/nova
max_pool_size = 4
[api_database]
connection = mysql+pymysql://nova:<*redacted*>@127.0.0.1/nova_api
max_pool_size = 4
[glance]
api_servers = https://10.113.1.62:9292
[neutron]
url = https://10.113.1.56:9696
region_name = RegionOne
auth_url = https://10.113.1.57:35357
auth_type = password
project_domain_name = service_domain
user_domain_name = service_domain
project_name = services
username = nova
password = <*redacted*>
# True
service_metadata_proxy = True
metadata_proxy_shared_secret = <*redacted*>
[keystone_authtoken]
auth_type = password
auth_uri = https://10.113.1.57:5000/v3
auth_url = https://10.113.1.57:35357/v3
project_domain_name = service_domain
user_domain_name = service_domain
service_type = compute
project_name = services
username = nova
password = <*redacted*>
signing_dir = /var/cache/nova
memcached_servers = inet6:[::1]:11211
[cinder]
os_region_name = RegionOne
cross_az_attach = True
[osapi_v3]
enabled=True
[conductor]
workers = 4
[oslo_messaging_rabbit]
[oslo_messaging_notifications]
driver = messagingv2
transport_url = rabbit://nova:<*redacted*>@10.113.1.58:5672/openstack
[notifications]
notification_format = unversioned
[oslo_concurrency]
lock_path=/var/lock/nova
[vnc]
[spice]
[serial_console]
enabled = false
base_url = ws://10.113.1.59:6083/
[placement]
auth_url = https://10.113.1.57:35357
auth_type = password
project_domain_name = service_domain
user_domain_name = service_domain
project_name = services
username = nova
password = <*redacted*>
os_region_name = RegionOne
region_name = RegionOne
randomize_allocation_candidates = true
[scheduler]
# NOTE(jamespage): perform automatic host cell mapping
# until we can orchestrate this better
# using the nova-cc <--> nova-compute
# relation
discover_hosts_in_cells_interval = 30
workers = 4
[filter_scheduler]
enabled_filters = AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,SameHostFilter
# Disable BuildFailureWeigher as any failed build will result
# in a very low weighting for the hypervisor, resulting in
# instances all being scheduled to hypervisors with no build
# failures.
# https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1818239
build_failure_weight_multiplier = 0.0
[api]
auth_strategy=keystone
[wsgi]
api_paste_config=/etc/nova/api-paste.ini
[pci]
[oslo_middleware]
# Bug #1758675
enable_proxy_headers_parsing = true
[quota]
Thanks in advance for any help!
Looking at your configuration file it makes me suspicious why no configurations are made in the
[vnc]section.It should look like this, for example:
IP_ADDRESSbeing the management IP address of the computer you are running nova-compute on.My advice:
Change your configuration like explained above, restart the nova services and then test via
which URL is returned and whether you can access this URL using your browser.