German Companies Face Challenges in AI Governance Amid Shift from Public Cloud to On-Premises

German companies are increasingly adopting artificial intelligence (AI) technologies but are grappling with significant governance and infrastructure challenges. Recent studies reveal a notable trend of businesses moving AI workloads away from public cloud environments back to private or on-premises infrastructures, motivated by concerns over data sovereignty, security, performance, and costs.

A survey by Cloudian involving 203 IT decision-makers found that nearly 79% of companies have already relocated or are in the process of relocating AI workloads from the public cloud to private infrastructure. Furthermore, 91% prefer on-premises or private cloud solutions for AI applications that handle sensitive data. Concerns over “Shadow AI”—the unregulated use of cloud-based AI tools by employees—were highlighted by 74% of respondents as significant security risks. Half of the companies have implemented access controls following incidents where confidential data was uploaded to external tools like ChatGPT.

Cost overruns further fuel this infrastructure shift: 40% of businesses reported cloud AI spending exceeding budgets by over 30% in some cases. Performance also matters greatly, with 75% emphasizing that on-premises setups are necessary for acceptable AI workload performance due to network latency. While hybrid infrastructures are expected to gain popularity, the public cloud still plays an important role in AI deployments. Most companies (86%) anticipate increasing their AI budgets in 2026, with 40% expecting rises of at least 25%.

Despite growing AI use, governance maturity remains a challenge. A study by Red Hat including 500 IT decision-makers across Europe indicated that only 30% of German companies have mature AI governance frameworks. Additionally, just 57% maintain exit strategies for AI providers in case of access restrictions, with 37% acknowledging that switching providers would moderately to severely impact business continuity. On a positive note, 51% of companies have full transparency over where their AI data is stored and processed, giving Germany a high transparency score of 97% relative to other European countries. Open-source solutions are seen by 69% as a potential way to enhance AI governance and control.

Red Hat is addressing sovereignty concerns by offering services to promote European digital sovereignty and tools to evaluate companies’ digital sovereignty.

In summary, German businesses are navigating a complex landscape balancing AI adoption with governance and infrastructure challenges. The shift toward on-premises and private AI workloads underscores priorities around data control, security, and performance, while governance frameworks and provider exit strategies still require development to ensure sustainable and secure AI integration.