The idea

Audac stands for Automotive Datacenter.

The goal is to provide datacenter like infrastructure to air gapped systems, like cars or mobile machines.

This includes PKI, updateability, extendability, zero-trust, recoverability, etc.

The primary target is to provide the basic infrastructure for prototype fleets.

This repository is a proof of concept to provide a platform for datacenter like infrastructure to automotive use cases, especially for development.

Context

In the last 15 years, datacenter related domains have created a vast amount of tools, procedures and best practices for robust and scalable infrastructure.

Tools like Kubernetes (k8s), S3 storage, Terraform and PostrgreSQL have revolutionized scalability and maintainability of infrastructures data and applications. Standards like PKI (public key infrastructure), VPN, OpenAuth2 and secret vaults allow for secure zero trust architecture.

And the raise of AI tools enables capabilities that were unthinkable a few years ago.

This project aims to make all these available in air gapped systems like automobiles.

Target users

Prototype vehicles are typically owned by several different teams. Each of them has a different pain point:

• Prototype vehicle teams want vehicles that stay in a known, reproducible state. They care about fast reflashing, configuration by vehicle identity, and recovery after a test drive breaks something.
• Data and validation teams want logs and traces to leave the vehicle reliably. They care about storage and data offload.
• IT security wants proof that a stolen or lost prototype leaks nothing. This is addressed by PKI, zero trust and encryption at rest.
• Platform and tooling teams are the likely adopters of a fleet infrastructure platform. They often build in-house and are skeptical of external stacks.

First use cases

The following use cases have priority:

• Fleet identity and trust. Every vehicle gets its own identity and CA. IT security can show that a lost prototype does not leak anything.
• Running newly developed functions. Developers deploy software to the vehicle like they would to a datacenter. Recording data is part of this use case, because it depends on which data is recorded by the project.

The first user is a project of an OEM that requires data recordings on multiple cars. This is around 10 cars, recording mostly camera and bus data.

The first demo and the success criteria are not defined yet.

Design principles

Audac is an open tech stack. It aims to support many solutions from the start, not the solution of a single customer.

• The core stays small. It provides the system cluster, PKI, storage, secure upload and the AI harness.
• Customer specific features are built on the application side, not in the Audac system. Recording is an example.
• A feature enters the core only when at least two customers need it.
• The stack is developed incrementally together with interested OEMs. The result serves as a guideline for implementing custom features.

Requirements

This PoC has the following primary requirements

• All features must be available as an air-gapped system, that is everything must work under the assumption that no access to remote infrastructure is available. As a consequence, no remote AI services can be used.
• The system must be updateable over the air (OTA) or via local update.
• The system must provide a PKI infrastructure for other in-vehicle systems on the local network.
• The system must be able to identify itself and receive specific configurations.
• It must be assumed that rogue parties have access to the system physically. This requires a zero trust architecture on all levels, especially at rest.
• IT security is the primary driver. Where requirements conflict, security takes precedence. The security design must be reviewable by IT security teams. This includes a documented threat model, the storage of keys and the protection of the root CA. Relevant automotive standards, such as TISAX, ISO/SAE 21434 and UNECE R155/R156, are used as a checklist.
• The system must be maintainable remotely, even when components are not operationable. This excludes cases when no data connection is possible.
• The system must support multiple concurrent data connections (e.g. mobile, wifi, satellite, etc).
• The system must work autonomously until it has connectivity to the infrastructure of the project, via wifi, mobile or satellite. Audac provides storage via Kubernetes and manages the secure upload of logs, recordings and similar data to the infrastructure of the project. How the data is recorded is part of the project, not of Audac.
• The system must provide a Agentic harness with local inference.
• A UI shall be available for live monitoring of the system and tracing problems.

Architecture decisions

Kubernetes

The backbone of the internal architecture is Kubernetes. Multiple distrbutions are candidates, but the k3s is the most promising, because of its low resource requirements.

Linux as operating system

The operating system must be a Linux distribution, potentially Minimal Ubuntu, because it is very popular among engineers.

PKI

Audac does not operate its own root CA. Its CA certificate is derived from a CA certificate of the OEM. Certificates issued by Audac are therefore trusted by existing OEM tooling.

The CA hierarchy has the following levels:

1. OEM CA.
2. Project CA. It is derived from the OEM CA, one per project. It can create vehicle CAs, but no further CA levels (path length of one).
3. Vehicle CA. It is derived from the project CA, one per vehicle. This is the CA certificate Audac uses in the vehicle.
4. End entity certificates, for example for other in-vehicle systems on the local network.

Audac must not be able to create sub CAs. The vehicle CA is constrained accordingly (path length of zero). Audac can only issue end entity certificates.

The project CA and its infrastructure are outside the scope of Audac. Audac's scope is the vehicle. Providing the project CA is a prerequisite that IT security of the project must fulfill. The project CA lives on the platform of the project, for example in the cloud environment of the project. Its key must be protected by a dedicated HSM service, for example Azure Managed HSM, depending on the platform of the project.

A separate CA per vehicle limits the impact of a lost or stolen prototype to a single certificate, which can be revoked. A project CA allows a project to enroll and revoke its vehicles without involving the OEM for each vehicle.

Certificate expiration

The vehicle CA has an end date. Audac must never keep operating beyond the end date of its certificate.

Different scenarios can be implemented for reaching the end date:

• Stop the system startup until the certificate is renewed.
• Erase the system when the end date is reached. This is the paranoid scenario.

Clock manipulation detection

Reaching the end date requires a trustworthy clock. The following options are available for detecting clock manipulation, intentional or unintentional. They are not final decisions. Solutions require close cooperation with the users.

• Check the TPM counter to detect clock resets. This is the minimum safeguard.
• Persist the clock continuously in the Audac system to detect clock tampering.
• Listen to the clock of the vehicle.
• Include a mandatory GNSS receiver in Audac.
• Include an atomic clock receiver.

This follows a swiss cheese approach. Each single option can be hijacked, but hijacking all of them together becomes more and more difficult.

Dedicated hardware

Audac runs on its own dedicated compute hardware. This is intentional. It serves the system under development, for example for data recording and for running newly developed functions. It is separate from the series compute of the vehicle.

KVM/QEMU for virtualization

Runtimes are separated by virtual machines running on KVM/QEMU. The host is a minimal Linux distribution. At least two Kubernetes clusters are used, a system cluster and a user cluster. A range of virtual machines with varying performance and resources must be provisioned for them.

Proxmox was considered and dropped for now, because of its larger attack surface, its own host operating system and its overhead on compact hardware. Functions a management layer would provide, such as the lifecycle of virtual machines, snapshots, backups and recovery, and updating the host, are not solved yet. See the open problems below.

OpenBao and TPM2

OpenBao runs in the system cluster. The user cluster reaches it through egress, ingress and routing rules. OpenBao is assumed to be a secure product, which is safe to use when access rules are defined correctly.

Critical credentials are stored in a TPM2 module. A TPM2 module also stores the disk encryption keys for encryption at rest. A TPM failure is acceptable from the standpoint of Audac. In this case the whole setup must be replaced.

For the PoC, the vehicle CA is stored in OpenBao. This is a known deviation from the requirement to protect against rogue parties with physical access. If more hardening is required, a TPM or an HSM module can be considered later.

Recording nodes

Audac provides storage via Kubernetes and manages the secure upload. How data is recorded is part of the project.

An optional architecture adds nodes on dedicated recording machines that connect to the Audac cluster.

AI harness

The AI harness is part of the Audac system, so that it is integrated into the security model. This is an exception to the rule that only features needed by at least two customers enter the core, and the reason is security.

The AI model is a custom feature on the application side. The only requirement is that the model supports Ollama. The harness provides a clean interface between the model and the system. All inference runs locally.

Audac may ship with a ready to go AI model. It can be changed via the settings of the harness. This default is a reference model. It is replaceable and not a supported feature. Its release cycle is separate from the core.

Monitoring UI

A monitoring UI runs in the vehicle. It works without connectivity and serves live monitoring of the system and tracing of problems.

Monitoring the complete fleet is not part of Audac. Audac can provide the data via upload, but the fleet view is a problem of the OEM and a prerequisite that the OEM must fulfill.

Single sign-on

Audac provides a Keycloak instance for single sign-on. It can be configured to sync with an upstream directory. The sync is one-way, downstream from the directory to the vehicle.

Networking

The network path and its policy belong to the OEM. Which links are allowed, what data may travel over which link and which IT security rules apply differ between organizations. They are configuration, not decisions of Audac.

Audac provides the vehicle-side mechanism:

• The vehicle connects out and authenticates with its vehicle CA.
• Multiple uplinks can exist, and the system copes with them coming and going.
• Data is kept until a link exists and uploads resume after a drop.

Audac documents what it expects from the network.

Open problems

This section documents potential problems, not solutions. Solutions require close cooperation with the users.

Trusted time

• The vehicle is offline, so it depends on a local clock. An attacker with physical access may set the clock back, and a failing RTC battery may reset it unintentionally.
• A TPM2 clock only advances while the TPM is powered. A parked vehicle does not measure the time it was switched off.
• Restoring an old disk image also restores an old "last seen" time.
• GNSS can be spoofed or jammed and is not available in garages or tunnels.
• The vehicle clock is often set by test tools and may not be trustworthy.
• Different time sources can disagree, either because of an attack or because of a hardware fault. It is not defined which source wins.

Certificate expiration and revocation

• A vehicle that does not start cannot be renewed remotely. This conflicts with remote maintainability.
• An erase at the end date can destroy valuable test data through a clock error, a failed renewal or a long parking period.
• Vehicles that are offline for weeks cannot receive revocation information. It is not defined how they treat outdated revocation information.

Enrollment and identity

• The first issuance of a vehicle CA needs a trustworthy path. It is not defined who approves it and how.
• Prototypes often have no stable identity, such as a VIN, at the beginning.
• Prototype hardware is swapped frequently. A replaced TPM or board changes the identity of the vehicle.
• A chain of five CA levels may exceed limits of embedded TLS stacks.

Keys and encryption at rest

• A disk key sealed in the TPM does not protect a stolen, complete vehicle computer that boots and unlocks itself without further checks.
• The bus of a discrete TPM can be sniffed by an attacker with physical access.
• OpenBao must be unsealed on every start. Unattended restarts and protection of the unseal key can conflict.
• A compromised project CA affects all vehicles of the project.
• The vehicle CA is stored in OpenBao for the PoC. If the machine boots and unlocks itself, someone with access to the running system can reach the CA key.
• A hardware TPM can be passed through to only one virtual machine. It is not defined which one owns it and how other clusters request operations without accessing it directly.
• A software TPM in a virtual machine gives no hardware protection.

Dedicated hardware

• Audac is only useful if it connects to the system under development, for example via CAN, automotive Ethernet, cameras, radar or lidar. Each of them requires hardware, drivers and timing support.
• Data recording at this bandwidth can reach terabytes per day. Storage and offload are hard problems.
• Audac connects to the vehicle networks and can therefore become a path into them. It is not defined how it is segmented from the vehicle buses and what an attacker gains by compromising it.
• A prototype is switched on and off abruptly, has voltage dips, gets hot and vibrates. This threatens the file system, the key store and the encrypted disk.
• Recording from many sources needs consistent timestamps.
• Dedicated hardware in every prototype must be cheap and small enough to be accepted by teams.
• Several teams may run newly developed functions on one Audac. It is not defined how they are isolated from each other.

Virtualization

• Without a management layer, the lifecycle of virtual machines has to be provided by Audac.
• Snapshots, backups and recovery of virtual machines are not solved. A broken virtual machine should not require a workshop visit.
• Updating the host and the hypervisor over the air is not solved.
• A hypervisor and at least two Kubernetes control planes consume resources on compact hardware.
• Data recording, vehicle interfaces and AI inference need PCIe or GPU passthrough to virtual machines. This constrains the choice of hardware and may affect timing of the recording.
• The trust boundary between the system cluster and the user cluster is only defined by the rules that let the user cluster reach OpenBao. Ingress, egress and routing rules are only as strong as their enforcement. It is not defined who writes and reviews them, and whether administrators of the user cluster can change them.
• OpenBao access rules are only as good as their definition. A policy that is too broad or a leaked token defeats a secure product.
• User workloads depend on OpenBao, which must be unsealed on every start. It is not defined what happens to the user cluster while OpenBao is unavailable.
• A workload in the user cluster can flood OpenBao with requests. It is not defined where access logs are stored and who reviews them.

Recording nodes

• Each recording node needs an identity and must be allowed to join the cluster. It is not defined who approves a new node and how a swapped or added node is enrolled.
• A recording node sits in the vehicle and can be reached physically. It is not defined what an attacker can reach in the cluster after taking it over.
• It is not defined whether recording nodes join the system cluster or the user cluster.
• It is not defined which node uploads, and how recorded data moves between nodes at high data rates.
• Data from several machines needs a common time base.
• More machines in the vehicle need more power, space and cost. This weakens the argument of a compact single box.

AI harness

• It is not defined whether Ollama is the runtime or the interface. As a runtime, it ties Audac to its hardware support and its performance on embedded GPUs.
• It is not defined whether the harness is an agent runtime with tool calling or a gateway to local inference.
• What an agent can reach is not specified, for example recordings, vehicle buses and the upload path. Prompt injection through recorded data or logs is a possible attack path.
• It is not defined which cluster gets the GPU and whether recording and inference can share it.
• A loaded model, recording and two clusters compete for memory, power and cooling on compact hardware, at a cost per vehicle.
• Models are large files. Updating them over a weak link is slow, and a tampered model is an attack. Integrity, source and license of models are not handled.
• AI output must never feed a safety relevant function. It is not defined how this boundary is communicated to customers.
• The first AI use case is not defined. Without it, the default model cannot be chosen to serve a purpose.
• Customers will judge the AI feature by the quality of the default model.
• Many models restrict commercial use or redistribution. The license of the default model is not defined.
• A default model must run acceptably on compact hardware and enlarges the image, which makes updates over a weak link slower.
• It is not defined who maintains the default model, when it is replaced and who tests it.
• Changing the model via harness settings is a security relevant action. It is not defined how it is authorized, checked for integrity and logged.

Monitoring UI

• The primary user of the first version is not defined. Test engineers, the Audac team, IT security and project developers need different views.
• The UI is another attack surface on the vehicle network and can show sensitive data, such as recordings or camera previews.
• Application side features, such as recording status, need a way to appear in the UI without changing the core.
• Logs, metrics and traces use memory, CPU and disk on compact hardware. It is not defined what leaves the vehicle.
• Telemetry is small, but it should reach the OEM first on a short link. It is not defined how it is prioritized.
• The Audac team maintains the system, but the fleet view belongs to the OEM. Diagnosis of a failing vehicle depends on infrastructure Audac does not control.
• An OEM project may expect a fleet dashboard to come with the product.

Single sign-on

• Synced accounts become stale in a vehicle that is offline for weeks. A person who leaves stays valid until the next sync. It is not defined how old the synced data may get.
• A directory sync puts names, groups and password hashes into a vehicle that can be stolen. It is not defined what is synced.
• Tokens and sessions depend on the clock. The trusted time problem also affects logins.
• Keycloak needs memory and a database on compact hardware. Lighter alternatives are not compared yet.
• Single sign-on is a new critical component. Its admin account, secrets, start order and availability are not defined. If it is down, nobody can log in to diagnose the vehicle.
• It is not defined how access works when single sign-on itself is broken, and how this emergency access is secured.
• The identity team of the OEM must approve a directory sync into a vehicle.
• It is not defined whether single sign-on serves everyone, or only the Audac UI and system components, with applications bringing their own login.

Connectivity and data upload

• Recordings can reach terabytes per day. Wifi at a workshop, mobile in the field and satellite are very different in bandwidth and cost. It is not defined what is uploaded over which link.
• Connectivity windows can be short. It is not defined which data is uploaded first, for example logs, traces or full recordings.
• Links drop frequently. Interrupted uploads must resume without corrupting or duplicating data, and the project side must know which data is complete.
• Without connectivity the disk fills up. It is not defined what is overwritten or refused, and who decides.
• Data is at risk until the project infrastructure has confirmed it. This includes the loss of all data on the disk when a setup is replaced. It is not defined when data may be deleted.
• The vehicle must prove its identity to the project infrastructure. It is not defined how this is verified, and what happens when the certificate has expired or is revoked.
• Camera recordings can contain people and license plates. This brings legal requirements for storage, transfer and retention.
• Uploads often happen while the vehicle is parked. Power state and battery drain during an upload are not defined.
• It is not defined how data leaves a vehicle that never has connectivity, for example on a test track without coverage.

Networking

• Mobile carriers usually block inbound connections. Remote maintenance needs the vehicle to connect out. It is not defined when it connects, especially while parked.
• The external connection is another attack surface. Its separation from the vehicle buses is not defined.
• Network time is not trustworthy. This affects the trusted time problem.
• Real link behavior is hard to reproduce in a lab.

Scope and adoption

• No threat model is documented yet.
• Other in-vehicle systems have their own roots of trust. Prototype systems often run with development keys or open debug ports, which Audac cannot change.
• It is open who maintains an open source Audac and who supports prototype fleets when they fail.
• The relationship to existing initiatives like Eclipse SDV is undefined.

To be defined

Open points are tracked as a checklist. An item is checked once it is solved.

• Recording gap: who provides recording for the first project, and where the responsibility of Audac ends.
• Responsibilities: who maintains the system cluster and who maintains the user cluster.
• Support boundary: who is responsible when a project workload breaks the system.
• Support capacity and service level for the first project.
• Funding model and licensing: how OEMs fund incremental development, and who owns and licenses the result.
• Remote maintenance of vehicles that are offline for weeks.
• First demo and success criteria of the first project.
• Project CA prerequisite: who provides it and when.
• Threat model.
• Supported hardware, sensors and update policy.
• Extension points: interfaces between the core and applications, for example for storage, upload and identity.
• AI agent reach: what the agent can reach, for example recordings, vehicle buses and the upload path, is part of the specification.
• Telemetry data contract: format and content of the data the OEM needs to build a fleet view, for example health, certificate expiry, storage and upload status.
• Access of the Audac team to telemetry, because the fleet view belongs to the OEM.
• Network responsibilities: the uplink interface, who supplies connectivity hardware such as modems and satellite terminals, and what Audac requires from the network.

Responsibilities

Audac is not maintained by the project team, but by the Audac team. Ideally it just works as specified. The project deploys its workloads to the user cluster. It is not defined who is responsible when such a workload exhausts resources or breaks recording, and what the specification promises to the customer.

Recording gap

Audac provides storage and manages the secure upload. How data is recorded is part of the project. The first user, an OEM project, requires data recordings on multiple cars. The project may expect recording to work with Audac without building it themselves. Commercial data loggers already cover recording today. It is not defined who provides recording for the first project, and where the responsibility of Audac ends.