Build local AI, connect company knowledge

An AI knows a great deal about the world and nothing about your company. It does not know last year’s quotes, your maintenance reports, your quality manual, or what two long-serving colleagues carry in their heads about a particular machine. That very knowledge is what makes AI useful in day-to-day work.

So the question is not whether to make it available to an AI, but where. We help companies build high-quality AI solutions in-house or in a hybrid setup and make their knowledge available to them, aligned with the requirements of the GDPR and NIS2. This article shows how that works, using the permissions that already apply in your company. Hybrid means: confidential material stays on systems under your control, while non-critical material may run through a cloud service, according to rules you define.

Why the topic is pressing right now

Three figures, all from this year and last:

57 percent of companies with 20 or more employees now use AI. But 59 percent of them say they are not tapping its potential at all. (Bitkom, 14 September 2026)
In around four in ten companies, managers know or suspect that employees use private AI accounts for work. Only just over a quarter have officially provided generative AI. (Bitkom, 21 October 2025)
On average, 1.3 million people retire every year, while only around 800,000 enter the workforce. (IW Köln, 15 June 2026)

Screenshot of the Bitkom press release of 21 October 2025: 4 in 10 companies assume at least that private AI tools are in use; 26 percent provide access to generative AI

Bildschirmfoto: Bitkom, Presseinformation „Beschäftigte nutzen vermehrt Schatten-KI“, 21.10.2025, Ausschnitt, abgerufen am 30.09.2026.

The second figure means: employees use private AI accounts for work, and whatever they enter there sits outside the systems the company has approved. The third means: some of this knowledge is not written down anywhere and leaves with the people who hold it.

Like ChatGPT, but with your company knowledge: how it works

The technology behind it is called RAG, retrieval-augmented generation. The AI is not trained on your documents. Instead, for every question the system finds the matching passages in your content and gives them to the language model as the basis for its answer. The answer names its source.

This has a data protection advantage that the supervisory authorities point out themselves. In its guidance on RAG systems from October 2025, the German Data Protection Conference (DSK) writes: inside the language model itself you cannot control who may access what, but in the document base of a RAG system you can, using the familiar tools of a roles and permissions model. Entries there can be deleted selectively. And because the knowledge comes from your documents, a smaller model is often enough, one that can also run on your own servers in-house. This makes it possible to avoid personal data going to the providers of large online models.

Screenshot from the German data protection conference guidance on RAG systems, section 3.3 integrity and confidentiality: access cannot be controlled inside the language model itself, but it can in the vector database and reference documents through a rights and role concept

Bildschirmfoto: Datenschutzkonferenz, Orientierungshilfe zu generativen KI-Systemen mit RAG-Methode, Version 1.0, Oktober 2025, Abschnitt 3.3, Ausschnitt.

Your own knowledge system in five steps

Choose one use case, not all of them. For example the technical support team answering questions about equipment, or the sales office looking for similar quotes. A case you can assess after four weeks.
Review the sources. Which drives, which SharePoint areas, which wiki? Outdated price lists and duplicates should be removed first. An AI that quotes from a 2018 list loses its users’ trust on day one. Knowledge from experience that so far exists only in people’s heads gets written down beforehand, for instance in short interviews.
Carry over the permissions. More on this in the next section. This step decides whether the system becomes a tool or a data protection incident.
Start small and measure. Beforehand, it is settled who maintains the sources and who deletes what has to be deleted. Then a pilot with ten to fifteen people who get a short introduction and ask real questions. What counts is how often the answer is right and how often the source fits, not how impressive it sounds.
Set up operation and training for the long term. Who brings in new models, who reviews the sources regularly? And everyone who uses the system needs an introduction; the AI Act requires AI literacy measures anyway.

We are happy to work out with you which use case pays off first for you and which sources are suitable for it.

Arrange a consultation

Does the AI access every file? Only if you let it

A knowledge system that knows all documents also knows salary lists, personnel files and board minutes. Without an access control model, anyone can ask about them.

The solution is to carry over the existing permissions: anyone who may not open a file on the drive or in SharePoint will not see it in any answer either, once the permissions have been synchronised. There is software that does exactly this. Onyx, for example, synchronises permissions with SharePoint, including nested groups from Entra ID, in its paid edition. In planning, this is the most important point and the one most often skipped.

Two limits come with it. Permissions changed on the drive only reach the knowledge system at the next synchronisation; you decide how often that runs. And anyone with access to a document also gets summaries of it. Whatever is shared too generously on the drive is not protected by AI but found more easily. That is why sharing permissions should be reviewed before launch.

Every company can now have its own AI server

A smiling IT consultant in a navy blazer holds a small AI server with a gold-coloured metal front on her hand, with an open-plan office in the background

For a long time, local AI meant a server room full of graphics cards. That has changed. One example is NVIDIA’s DGX Spark:

Memory: 128 GB, shared by processor and AI
Size: 15 × 15 × 5 cm, 1.2 kg
Power draw: according to NVIDIA 38 watts idle, at most 233 watts
Model size according to NVIDIA: inference up to 200 billion parameters, two linked devices up to 405 billion
Example measurement: the open model gpt-oss-120b at around 60 tokens per second for a single request, around 40 with long context (llama.cpp, February 2026); less per person with several users at once
Price: from around 5,000 euros net at German retailers, up to just over 8,000 euros depending on manufacturer and SSD (as of the end of September 2026)

Screenshot of the NVIDIA DGX Spark specifications: NVIDIA Grace Blackwell architecture, 20-core Arm CPU, up to 1 PFLOP FP4 tensor performance, 128 GB LPDDR5x coherent unified system memory, 273 GB/s memory bandwidth, 4 TB NVMe storage

Bildschirmfoto: NVIDIA, DGX Spark, Spezifikationen, Ausschnitt, abgerufen am 30.09.2026.

The same internals are available from Dell, HP, Lenovo, ASUS, Gigabyte, MSI and Acer.

Two caveats come with it. First, the memory is fast enough for models that use only part of their parameters per word, such as gpt-oss-120b. Large models that always compute all parameters run considerably slower on it; NVIDIA itself measures around 4 tokens per second for Llama 3.3 with 70 billion parameters. Second, for a department what counts is not the speed for a single user but the load of many simultaneous questions. It should be measured before buying, not estimated.

Alternatives are computers with AMD’s Ryzen AI Max+ 395 (up to 128 GB), the Mac Studio with M5 Ultra (up to 512 GB, this configuration is announced for the end of October) and, for larger loads, classic servers with graphics cards such as the RTX PRO 6000 (96 GB per card). All have become more expensive in 2026 because memory chips are scarce; the DGX Spark Founders Edition, for example, rose from 4,180 to 4,800 euros in February.

Which model runs on it is also a licensing question. Freely usable under Apache 2.0 are, among others, gpt-oss from OpenAI, Qwen 3.5 from Alibaba and Gemma 4 from Google. With Meta’s Llama 4, the rights to the multimodal models are excluded for companies headquartered in the EU. And the widely used interface Open WebUI has been under its own licence since April 2025, which attaches branding conditions once there are more than 50 users.

Copilot and data protection: does everything stay in the EU?

For many, Microsoft 365 Copilot is the obvious route, and for content that already lives in Microsoft 365 it is often the right one. Two settings are worth knowing, both from Microsoft’s own documentation:

Flex routing: During peak load, requests may be processed outside the EU Data Boundary, in the USA, Canada or Australia. According to Microsoft, the data continue to be stored in the EU, except for limited pseudonymised data. For eligible tenants in the EU created after 25 March 2026, flex routing is switched on by default; for older tenants Microsoft asks administrators to review the setting.
Anthropic models in Word, Excel and PowerPoint: This processing takes place outside the EU Data Boundary. This setting, too, is on by default for tenants in the EU created after 25 March 2026. It is located in the Microsoft 365 admin center under Copilot, Settings, View all, as the setting for AI providers operating as Microsoft subprocessors. If you do not allow Anthropic there, these models cannot be used in Word, Excel and PowerPoint.

Screenshot of Microsoft Learn, Flex routing (EU and EFTA): flex routing is on by default for eligible tenants created after March 25, 2026; administrators of older tenants are encouraged to check the setting in the Microsoft 365 admin center

Bildschirmfoto: Microsoft Learn, „Flex routing (EU and EFTA)“, Stand 29.09.2026, Ausschnitt, abgerufen am 30.09.2026.

How to switch off flex routing:

Sign in to the Microsoft 365 admin center with an account that has the AI Administrator role.
Open Copilot, Settings, View all and select the flex routing setting for peak load times.
Choose the option that does not allow flex routing, and save.

Screenshot from the Microsoft 365 admin center: the setting flex routing during peak load periods with the two options allow flex routing during peak load periods and do not allow flex routing

Bildschirmfoto aus dem Microsoft 365 Admin Center, 30.09.2026: Copilot, Einstellungen, Alle anzeigen, „Flexrouting während Spitzenlastzeiten“. Die zweite Option schaltet Flex Routing ab; die Sprachmodelle von Copilot rechnen dann laut Microsoft auch bei Lastspitzen in der EU Data Boundary. Die Anthropic-Modelle regelt eine eigene Einstellung.

If you use Copilot, check both switches. If you have data that must not leave the company at all, you need a system in-house for that data. In practice it is often both: Copilot for breadth, a local system for the confidential core.

We review the Copilot settings of your tenant and change them for you on request: Get in touch.

What the AI Act says about it

Owing to the so-called Digital Omnibus, the rules for high-risk AI under Annex III, for example in recruitment or credit checks, only apply from 2 December 2027, and for high-risk AI in products under Annex I from 2 August 2028. Two obligations, however, already apply:

AI literacy under Article 4, since 2 February 2025. Under the wording amended by the Omnibus, providers and deployers of AI systems take measures to support the development of AI literacy of their staff. A deployer is any company that uses AI under its own responsibility. The obligation has remained, even if it is worded more softly.
Transparency under Article 50, since 2 August 2026, for example disclosing that you are talking to an AI.

Screenshot from EUR-Lex, Regulation (EU) 2026/1744: the new wording of Article 4 AI literacy. Providers and deployers of AI systems take measures to support the development of AI literacy of their staff

Bildschirmfoto: Verordnung (EU) 2026/1744, Artikel 1 Nummer 5, auf EUR-Lex, Ausschnitt, abgerufen am 30.09.2026.

An internal knowledge system that answers questions about manuals is, as a rule, not a high-risk system. AI literacy measures are still required, such as an introduction, and they belong in step 5. We train users and administrators for this.

An NMS trainer giving a talk on intelligent technology for businesses

What day-to-day operation costs: a sample calculation

This is a calculation based on assumptions, not a measurement. It shows only the running costs of a single device. Setup, permission synchronisation and licences come on top and depend on the project.

DGX Spark-class device: 5,000 euros net over 36 months, around 140 euros per month
Electricity: 150 watts on average (between 38 and 233 watts according to NVIDIA), 0.30 euros per kWh, around 33 euros per month
Operation and maintenance: 4 hours at 100 euros, probably a lower-bound estimate, 400 euros per month
Setup, permission synchronisation: depending on sources and use case, not quantified
Knowledge software licences: depending on the product, free in some cases, not quantified
Total, excluding the two uncosted items: around 570 euros per month

The largest item is not the device but its operation. Not included are setup, licences for the knowledge software, a second device as a fallback, and the question of whether one device can carry the load of your users. The calculation compares costs, not performance.

What we do

Two NMS colleagues going through content on a tablet together

We set up high-quality AI solutions on your behalf, if you wish as AI hosting in Germany in our data centre, on your premises or in your Azure environment, with sign-in through your Microsoft 365 account. You decide who uses the system for what, and are therefore the deployer within the meaning of the AI Act; we take care of the technology, updates and maintenance. We start with the use case and the sources, not with the hardware. And if it turns out that Copilot is enough for your data, we will tell you that too.

Talk to us, by phone at +49 7731 969390 or by email at info@nms.de.

Get in touch

More on our page about self-hosted AI and in the article AI on your own premises.

Sources

All sources accessed on 30 September 2026.

Building local AI in your company and connecting your company knowledge