عملت revoke للـ token وفضل شغال! لما تشوف 401 أو 403 تدور فين؟ | Deep Dive 1

عملت revoke للـ token، وKeystone رفضه، بس Nova فضل قابله. في الحلقة دي بنعرف ده بيحصل ليه، وبنمشي ورا الـ token خطوة بخطوة: فيه إيه وإيه اللي مش فيه، مين عمله ومين عمل له validate من الـ logs، وليه الـ Fernet keys لازم تبقى هي هي على كل الـ controllers. وبعدين 3 كسرات: token اتعمله revoke وفضل شغّال 5 دقايق، و401 بتيجي وتروح لما الـ keys تختلف بين الـ controllers، والفرق بين 401 و403. الـ checklist اللي تمشي عليها لما تشوف 401 أو 403 في الشغل موجودة في أول كومنت. 🇬🇧 English subtitles available, turn on CC. I revoked a token. Keystone rejected it, but Nova kept accepting it. We follow one token step by step: what's inside it, which controller issued it and which one validated it (from the logs), and why the Fernet keys must match on every controller. Then 3 breaks: a revoked token that keeps working for 5 minutes, 401s that come and go when the keys differ, and 401 vs 403. Chapters 00:00 I revoked the token. Why does Nova still accept it? 02:55 Intro 03:43 What's in a token? Scope and domain 06:32 How long a token lives, and changing its expiry 11:22 Fernet: what's inside a token 14:06 Reading the token header by hand 18:28 Which controller issued it, which one validated it? 22:19 Same keys on every controller (bootstrap, push, rotate) 27:20 Who does the CLI talk to? Endpoints, public vs internal, ports 31:59 Where did the request stop? (--debug) 32:57 Does Nova open the token? 36:32 Break 1: why revoke didn't stop it 39:20 Break 2: why 401s come and go 45:00 Break 3: 401 or 403? 48:55 The 401/403 checklist 49:39 Recap and next: Glance The lab Kolla-Ansible 2025.1 · 3 controllers (HA, VIP) · 3 computes · 3 Ceph nodes · Fernet tokens Commands used in the episode cat token curl -i -H "X-Auth-Token: $(cat token)" http://10.0.0.100:8774/v2.1/servers openstack token revoke $(cat token) curl -I -H "X-Auth-Token: $(cat token)" -H "X-Subject-Token: $(cat token)" http://10.0.0.100:5000/v3/auth/tokens openstack role assignment list --user admin --project admin --names openstack token issue ssh ctrl01 grep expiration /etc/kolla/keystone/keystone.conf grep fernet_token_expiry /etc/kolla/globals.yml kolla-ansible reconfigure -i $INVENTORY -t keystone openstack token issue -f value -c id | tee token cut -c1-12 token | basenc --base64url -d | od -An -tx1 printf "%d\n" 0x$LAST_4_BYTES date -d @$SECONDS_SINCE_1970 ssh ctrl01 docker exec keystone grep -a -e $DD/Mon/YYYY:HH:MM:SS /var/log/kolla/keystone/keystone-apache-public-access.log ssh ctrl01 md5sum /var/lib/docker/volumes/keystone_fernet_tokens/_data/* cd /root/openstack-env/share/kolla-ansible/ansible/roles/keystone/tasks grep -n -A5 fernet_setup bootstrap_service.yml grep -n -A2 fernet-push distribute_fernet.yml openstack endpoint list -c "Service Name" -c Interface -c URL --sort-column "Service Name" grep OS_INTERFACE /etc/kolla/admin-openrc.sh grep vip_address /etc/kolla/globals.yml openstack --debug server list |& grep HTTP/1.1 ssh ctrl01 docker exec nova_api ls /etc/keystone/fernet-keys ssh ctrl01 grep memcached_servers /etc/kolla/nova-api/nova.conf ssh ctrl01 grep -c token_cache_time /etc/kolla/nova-api/nova.conf ssh ctrl01 ls /var/lib/docker/volumes/keystone_fernet_tokens/_data ssh ctrl01 docker exec -u keystone keystone_fernet keystone-manage fernet_rotate curl -sI -H "X-Auth-Token: $(cat old-token)" -H "X-Subject-Token: $(cat old-token)" http://10.0.0.11:5000/v3/auth/tokens ssh ctrl01 docker exec keystone_fernet fernet-push.sh openstack --os-password wrong-password token issue openstack project create demo openstack user create --project demo --password $PASSWORD $USER_NAME openstack role add --user $USER_NAME --project demo member openstack user list openstack server list -f json The 3 breaks 1) Revoke a token: Keystone says 401, Nova keeps saying 200 for up to 5 min (keystonemiddleware token_cache_time = 300 s) 2) Rotate the Fernet keys on one controller without a push: the same token gets 401 on ctrl01 and 200 on ctrl02. Fix: fernet-push.sh 3) Wrong password = 401 (no token at all). Right password, wrong role = 403 (policy says no) 💼 LinkedIn:   / ahmed-saey-eldin-224a28130   #OpenStack #Keystone #CloudComputing #TelcoCloud #DevOps