Global environment; Developer Portal and OAuth impaired
Incident Report for Akana
Beginning at 22:30UTC; the Gobal API Portal (Developler Portal) cluster was at reduced functionality.

Full functionality was restored by 02:56UTC.

Initial determination of the cause was due to multi-regional AWS EBS service issues impacting the io2 (provisioned IOPS) volume types, used by the Akana Mongo DB clusters. The EBS issues caused nvme kernel driver "panics", rendering the systems unresponsive. This resulted in increase API Portal java heap utilization and eventual exhaustion, and had a cascading affect across all the active API Portal JVMs with JVM Out of Memory conditions that induced rolling JVM restarts, and periodic loss of connectivity to the Developer Portals, and OAuth services.
Posted Oct 30, 2021 - 19:56 PDT