deployment appears stuck for 8+ hours, on Ubuntu 18.04 and minikube., also tried kind.

[outbackdingo]

deployment appears stuck for 8+ hours
NAME READY STATUS RESTARTS AGE
horizon-7f4f6d9fcb-rg9ks 0/1 Init:0/1 0 8h
horizon-db-init--1-mp7dm 0/1 Init:0/1 0 8h
horizon-db-sync--1-vlbq6 0/1 Init:0/1 0 8h
ingress-d5d8b9bdc-hh2wq 0/1 CrashLoopBackOff 155 (5s ago) 8h
ingress-error-pages-6d54c96946-bnlbl 1/1 Running 0 8h
keystone-api-85b965f486-47hzx 0/1 Init:0/2 0 8h
keystone-db-init--1-bf96g 0/1 Init:0/1 0 8h
keystone-fernet-rotate-27300960--1-q75ns 0/1 Init:0/1 0 129m
kupenstack-controller-678c66b58c-sxlwv 1/1 Running 0 8h
mariadb-ingress-56bbb59f78-mlncq 0/1 CrashLoopBackOff 183 (3m50s ago) 8h
mariadb-ingress-error-pages-77c55c9cd7-fvr6k 1/1 Running 0 8h
mariadb-server-0 1/1 Running 0 8h
memcached-memcached-9978cc66c-vvr7s 1/1 Running 0 8h
neutron-db-init--1-9wfm5 0/1 Init:0/1 0 8h
neutron-dhcp-agent-default-x7f5w 0/1 Init:0/2 0 8h
neutron-l3-agent-default-pj7ch 0/1 Init:0/2 0 8h
neutron-lb-agent-default-6qx2k 0/1 Init:0/3 0 8h
neutron-metadata-agent-default-44tbh 0/1 Init:0/2 0 8h
neutron-netns-cleanup-cron-default-4lkf8 1/1 Running 0 8h
neutron-server-656ccbf745-vts2f 0/1 Init:0/1 0 8h
nova-api-metadata-779bbbfd6b-qhdr4 0/1 Init:0/2 0 8h
nova-api-osapi-79bffdb64c-xbk8x 0/1 Init:0/1 0 8h
nova-bootstrap--1-vzqtb 0/1 Init:0/2 0 8h
nova-cell-setup--1-ljtcn 0/1 Init:0/2 0 8h
nova-conductor-955f5c4b-vm4kh 0/1 Init:0/1 0 8h
nova-consoleauth-545bd5cd69-kzzzf 0/1 Init:0/1 0 8h
nova-db-init--1-f7jms 0/3 Init:0/1 0 8h
nova-novncproxy-7f94c99b6f-znz2z 0/1 Init:0/3 0 8h
nova-scheduler-6fff8d5b86-stdsp 0/1 Init:0/1 0 8h
rabbitmq-rabbitmq-0 1/1 Running 0 8h

[Alt-Shivam]

Hey @outbackdingo
Can you please provide me with your environment info
So that I can rectify this issue.

[outbackdingo]

ubuntu 18.04 kubectl deployed kubernetes, rook-ceph

[outbackdingo]

i also just tried this with kind as per your document and it failed
kubectl get pods -n kupenstack
NAME READY STATUS RESTARTS AGE
glance-api-7649d4d869-b8qnk 0/1 Init:0/2 0 8m21s
glance-db-init-bnp9j 0/1 Init:0/1 0 8m21s
horizon-57b49b87d5-cghlx 0/1 Init:0/1 0 8m49s
horizon-db-init-2j5j5 0/1 Init:0/1 0 8m49s
horizon-db-sync-lv79c 0/1 Init:0/1 0 8m49s
ingress-d5d8b9bdc-2f9sl 1/1 Running 0 9m20s
ingress-error-pages-6d54c96946-vrphv 1/1 Running 0 9m20s
keystone-api-795996c9fd-7mzxq 0/1 Init:0/2 0 8m49s
keystone-credential-setup-pg77j 0/1 Completed 0 8m49s
keystone-db-init-dj6w5 0/1 Init:0/1 0 6m12s
kupenstack-controller-678c66b58c-drksl 1/1 Running 0 9m53s
libvirt-libvirt-default-g8stp 0/1 Init:0/1 0 9m19s
mariadb-ingress-556ff7f4ff-nstvl 0/1 CrashLoopBackOff 3 75s
mariadb-ingress-error-pages-7d9d8c67fd-htkcw 1/1 Running 0 9m20s
mariadb-server-0 1/1 Running 0 9m20s
memcached-memcached-9978cc66c-m9lbw 1/1 Running 0 9m21s
neutron-db-init-sb4lq 0/1 Init:0/1 0 8m49s
neutron-dhcp-agent-default-k5957 0/1 Init:0/2 0 8m50s
neutron-l3-agent-default-2h86s 0/1 Init:0/2 0 8m50s
neutron-lb-agent-default-vjdgf 0/1 Init:0/3 0 8m50s
neutron-metadata-agent-default-6fddm 0/1 Init:0/2 0 8m50s
neutron-netns-cleanup-cron-default-jwj2q 1/1 Running 0 8m50s
neutron-server-6fb5c6975f-mbs88 0/1 Init:0/1 0 8m50s
nova-api-metadata-5667c5769f-76nq6 0/1 Init:0/2 0 8m49s
nova-api-osapi-6d6ffcc55b-ht7n8 0/1 Init:0/1 0 8m49s
nova-bootstrap-fcfnd 0/1 Init:0/2 0 8m49s
nova-cell-setup-wnxdn 0/1 Init:0/3 0 8m49s
nova-compute-default-szhf4 0/1 Init:0/3 0 8m49s
nova-conductor-5d46cd5899-299zp 0/1 Init:0/1 0 8m49s
nova-consoleauth-5fdb5f769d-clbz7 0/1 Init:0/1 0 8m49s
nova-db-init-2wjzt 0/3 Init:0/1 0 8m49s
nova-novncproxy-654794b96d-mnb62 0/1 Init:0/3 0 8m49s
nova-scheduler-5b48bf85f-xmxlw 0/1 Init:0/1 0 8m49s
placement-api-747d998dcf-t6rr8 0/1 Init:0/1 0 8m50s
placement-db-init-hjl9s 0/1 Init:0/1 0 8m50s
placement-db-sync-nnz52 0/1 Init:0/1 0 8m50s
rabbitmq-cluster-wait-kkf87 0/1 Init:0/2 0 9m22s
rabbitmq-rabbitmq-0 0/1 Running 1 9m22s

[outbackdingo]

kubectl describe pod mariadb-ingress-556ff7f4ff-nstvl -n kupenstack
Name: mariadb-ingress-556ff7f4ff-nstvl
Namespace: kupenstack
Priority: 0
Node: kind-control-plane/172.19.0.2
Start Time: Sat, 05 Feb 2022 18:45:04 +0700
Labels: app.kubernetes.io/component=ingress
app.kubernetes.io/instance=mariadb-ingress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=mariadb
app.kubernetes.io/version=v10.2.31
application=mariadb
component=ingress
pod-template-hash=556ff7f4ff
release_group=mariadb
Annotations: configmap-bin-hash: bbdd6113686adcf6cd70ba5888f5eb864ff6bada7478807bc9631f9a8175db42
configmap-etc-hash: 319cef174bf71ffbbd4111b78c70f178ba4d7cc55365e688a764ed3cde40a519
openstackhelm.openstack.org/release_uuid:
Status: Running
IP: 10.244.0.36
IPs:
IP: 10.244.0.36
Controlled By: ReplicaSet/mariadb-ingress-556ff7f4ff
Init Containers:
init:
Container ID: containerd://62b4618b37ec06265538ae79453486fa17cca1cc5f849c2f5bf28d35cc2cb054
Image: quay.io/airshipit/kubernetes-entrypoint:v1.0.0
Image ID: sha256:c092d0dada614fdae3920939c5a9683b2758288f23c2e3b425128653857d7520
Port:
Host Port:
Command:
kubernetes-entrypoint
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 05 Feb 2022 18:45:06 +0700
Finished: Sat, 05 Feb 2022 18:45:08 +0700
Ready: True
Restart Count: 0
Environment:
POD_NAME: mariadb-ingress-556ff7f4ff-nstvl (v1:metadata.name)
NAMESPACE: kupenstack (v1:metadata.namespace)
INTERFACE_NAME: eth0
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
DEPENDENCY_SERVICE: kupenstack:mariadb-ingress-error-pages
DEPENDENCY_DAEMONSET:
DEPENDENCY_CONTAINER:
DEPENDENCY_POD_JSON:
DEPENDENCY_CUSTOM_RESOURCE:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tpmk4 (ro)
Containers:
ingress:
Container ID: containerd://bccd4e5cc6396571cefa0a79a10181e6714a7e7e60233f1f22a9723aec6fe3b2
Image: k8s.gcr.io/ingress-nginx/controller:v0.42.0
Image ID: k8s.gcr.io/ingress-nginx/controller@sha256:f7187418c647af4a0039938b0ab36c2322ac3662d16be69f9cc178bfd25f7eee
Port:
Host Port:
Command:
/tmp/mariadb-ingress-controller.sh
start
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Sat, 05 Feb 2022 18:46:52 +0700
Finished: Sat, 05 Feb 2022 18:46:52 +0700
Ready: False
Restart Count: 4
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: tcp-socket :3306 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: mariadb-ingress-556ff7f4ff-nstvl (v1:metadata.name)
POD_NAMESPACE: kupenstack (v1:metadata.namespace)
RELEASE_NAME: mariadb
INGRESS_CLASS: mariadb-mariadb-ingress
ERROR_PAGE_SERVICE: mariadb-ingress-error-pages
Mounts:
/etc/nginx/template/nginx.tmpl from mariadb-ingress-etc (ro,path="nginx.tmpl")
/tmp from pod-tmp (rw)
/tmp/mariadb-ingress-controller.sh from mariadb-bin (ro,path="mariadb-ingress-controller.sh")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tpmk4 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
pod-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
mariadb-bin:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mariadb-bin
Optional: false
mariadb-ingress-etc:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mariadb-ingress-etc
Optional: false
kube-api-access-tpmk4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: openstack-control-plane=enabled
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 2m30s default-scheduler Successfully assigned kupenstack/mariadb-ingress-556ff7f4ff-nstvl to kind-control-plane
Normal Pulled 2m29s kubelet Container image "quay.io/airshipit/kubernetes-entrypoint:v1.0.0" already present on machine
Normal Created 2m29s kubelet Created container init
Normal Started 2m28s kubelet Started container init
Normal Pulled 90s (x4 over 2m26s) kubelet Container image "k8s.gcr.io/ingress-nginx/controller:v0.42.0" already present on machine
Normal Created 90s (x4 over 2m26s) kubelet Created container ingress
Normal Started 90s (x4 over 2m26s) kubelet Started container ingress
Warning Unhealthy 90s (x2 over 90s) kubelet Readiness probe failed: dial tcp 10.244.0.36:3306: connect: connection refused
Warning BackOff 89s (x8 over 2m24s) kubelet Back-off restarting failed container