Machine Learning for Embedded Systems with ARM Ethos-U NPU

Train a tiny machine learning model in the cloud with TensorFlow, apply quantization to fit embedded resources, and deploy via TensorFlow Lite for microcontrollers, CMSIS-NN, Helium DSP, and ETHOS-U55.

Survey diverse machine learning architectures, from cnn-based and recurrent networks to transformers, mlps, gans, autoencoders, capsule and graph networks, and their limited embedded-device deployments.

Convert the trained TensorFlow model to a TensorFlow Lite file, apply quantization for embedded resources, and compile with the Vela compiler for the ARM NPU.

Explore how a cnn model uses pipeline stages—input, convolution, activation, pooling, flatten, fully connected layers, softmax—producing a probability distribution to recognize cats, dogs, or birds on embedded devices.

Explore the convolution stage by applying a 2x2 by 3 depth filter to RGB inputs (9 by 9 per channel) with stride to produce a 2x2 output, summed across channels.

After the convolution stage, the activation stage applies a rectified linear unit to introduce non-linearity in CNNs, computing max(0, x) and turning negatives to zero while keeping positives.

Pooling is an optional stage after activation that uses 2x2 windowing to reduce the convolution output, producing a single value; it is unnecessary for small inputs but helps memory fit.

Stack multiple layers by repeating convolution, activation and pooling to build cnn models that detect edges first and progressively recognize complex shapes and objects like dogs, cats, or birds.

Flatten converts multi-dimensional feature maps from convolution, activation, and pooling stages into a 1d vector, transforming a 4 by 4 by 3 matrix into a 1d vector.

Explore how a dense (connected) layer compares a flattened cnn output against reference images using 1d filters and biases to produce neuron outputs that form a probability distribution for recognition.

Apply softmax to logits from a dense layer to produce a probability distribution, then compare cat and dog probabilities to make a final decision with an optional threshold.

Train a model in TensorFlow, convert to TensorFlow Lite for microcontroller, apply quantization, and generate a firmware flow that targets M55, Helium DSP, and Ethos-U NPU via the Vela compiler.

Train a machine learning model within the TensorFlow framework to design and train a cnn or rnn, then save the model in h5 or TensorFlow saved model format.

Convert a TensorFlow h5 or saved model to a TensorFlow Lite flatbuffer format using the TensorFlow Lite converter, then apply quantization, clustering, and pruning to optimize for edge devices.

Parse and optimize a TensorFlow Lite flatbuffer model with the ARM Vela compiler, distributing operations to Ethos-U NPU, Helium DSP, or Cortex-M55 microprocessor based on hardware.

Vela compiles a stream of low-level commands for microcontroller, parsed by TensorFlow Lite and dispatched to M55 CPU, NPU, or DSP via ETHOS NPU driver and CMSIS neural network library.

The Vela compiler integrates hardware details into the TensorFlow Lite flat buffer workflow, producing low-level commands optimized for the embedded NPU, CPU, and DSP.

Explore how TensorFlow Lite models are optimized for embedded devices through pruning, clustering, and quantization, then compiled by Vela to generate hardware-specific commands and a ready-to-integrate TensorFlow Lite file.

Ensure Python 3.10+ is installed, along with pip3 for package management, and the Microsoft Visual C++ 14.2 build tools as prerequisites to install and use the Vela compiler.

Install the Arm Ethos-U Vela compiler by meeting prerequisites like Python 3.10 and Visual C++ tools, then install via pip3 or source method and verify with Vela version.

The Vela compiler exposes command line options to view version, generate micro operation reports, and configure ETHOS-U NPU or CPU via tuning and configuration files, with MACs up to 2048.

Explore how the Vela compiler system configuration file encodes memory layout, start addresses, memory area sizes, and input-output bandwidth to position descriptors parsed by the NPU hardware block.

Understand how the NPU uses four memory types—const, arena, cache, and arena cache size—to store weights, bias, and feature maps, and how the three SRAM modes govern data placement.

Learn how the Vela compiler tuning configuration file assigns operations to CPU or NPU blocks, and how kernel parameters, stripping, block splitting, and quantization routing modes shape low-level execution.

Explore how a optimized TensorFlow Lite file is generated by the Vela compiler, then quantized, clustered, and pruned for microcontroller firmware, and organized for NPU and DSP execution.

Explore the flat buffer file format used for TensorFlow models, including how to dump and read its content, define a schema, and access model commands efficiently without unpacking.

Dump and convert a TensorFlow Lite model to JSON using the flatc tool from Google, then explore the model structure with the resulting JSON files.

Explore the flat buffer file json presentation of a TensorFlow Lite model, detailing version, opcodes, buffers, and the subgraph with inputs, outputs, tensors, and operators.

Explain the TensorFlow Lite flat buffer opcode table, detailing built-in codes for CPU or DSP execution and custom codes for the Ethos-U NPU block.

Explain embedding input, weights, and bias buffers in the buffers section, use tensors to describe buffers, and create input and output placeholders to illustrate the ML model flow in firmware.

Explore how TensorFlow Lite flatbuffers organize subgraphs with inputs, outputs, tensors, and operators, detailing convolution stages, weights, bias, activation, and memory-efficient buffer reuse.

Explore how operator nodes index opcode buffers in flat buffer elements, using a lookup table to map to operations like convolution, custom, reshape, and activation.

Describe how flat buffer tensors indexing uses an indexing topology to map inputs, weights, and bias to tensor entries and a temporary output for convolution, improving readability of buffers.

Describe each buffer’s indexing, including buffer 2 for weights with dimension 3 by 3 by 1 by 8 and linked tensors, creating an intermediate stage between buffers and operators.

Vela uses memory layout information and detects the ETHOS NPU to optimize the model and insert ETHOSU_CONV_2D custom operations executed only by the NPU.

Learn how the Vela compiler creates low-level Ethos-U NPU operations, and how to dump input buffers and metadata to define input/output shapes, weights dimensions, and channel counts.

Explore how memory configuration, SRAM limits, and embedded data streams influence Ethos-U conv 2d npu operation, prompting weight matrix chunking and additional convolution stages.

Examine flat buffer metadata to reveal tensor buffer usage and sizes for ethos-u operators. Identify memory types, base address of sram, and temporary buffers that influence tensor offsets.

Explore TensorFlow Lite flat buffer metadata to assess overall memory usage and operator allocations, detailing memory types, input buffers, offsets, sizes, and ETHOS-U specific memory operations.

Extract architecture configuration metadata from tflite flatbuffer files for arm ethos-u npu, including npu type, memory layouts, bus width, interconnect, low-level operation names, and spatial mapping to npu execution blocks.

Inspect metadata appended to the flat buffer from the Vela compilation stage to reveal memory usage, memory types for low-level machine learning, low-level operations, tensor allocations, and related information.

Explore the ETHOS-U NPU input data stream, detailing low-level opcodes for convolutions and activations, DMA transfer commands, memory region addressing, and looping and tiling of tensors to fit microcontroller SRAM.

Explain how the NPU's DMA transfer low-level commands control reading and writing to memory, including addresses, sizes, and using NPU_SET_WEIGHT_REGION and NPU_SET_IFM_REGION for input tiles, weights, biases, and feature maps.

Describe how the NPU DMA start operation initializes transfers, sets input feature map and weight base addresses, polls for completion, loads data into NPU buffers, and stores outputs to memory.

Explain memory regions information for the NPU, including tensor buffers: inputs, outputs, weights, scratches; read-write compatibility, and coordination of CPU and NPU memory layouts to avoid race conditions.

Explore how looping and tiling logic breaks tensors into tiles to fit limited memory, with the Vela compiler generating DMA commands, loop control, and micro-operations for NPU convolution.

The Vela compiler computes tile sizes to fit memory, divides tensors into tiles, and emits DMA loads, stores, and NPU instructions for Ethos-U to process tiles for inference.

Explore NPU micro-operations and input configuration parameters, including opcodes for convolution, pooling, activations, and depthwise 2D convolution, and how the Vela compiler selects schemes like average or max pooling.

Explore ARM ethos-u NPUs, comparing high performance, high power units (N37, N57, N78, N77) with low power units (U55, U65, U85) for Cortex-A and Cortex-M.

Arm ethos-u npu enables low-power, high-throughput machine learning on select Cortex-M cores (M4/M7/M33 and newer), enabling object detection, sensor fusion, and real-time recognition.

Explore how the Cortex-M55 and Ethos-U55 NPU co-exist on a single system-on-chip, coordinating via the AMBA bus, interrupts, DMA, and shared memory workspace.

Explore how the Cortex-M55 and Ethos-U55 NPU share a system-on-chip through the NIC-400 bus interconnect, with tightly coupled memories, DMA interfaces, and prioritized arbitration for NPU over the CPU.

Explore how ARM Ethos-U NPU topologies connect Cortex-A and Cortex-M systems, including direct NPU to M, direct NPU to A, and A through M architectures, with power and use-case considerations.

Explore the Ethos-U NPU block diagram, from the DMA controller and control unit to local memory, weights decoder, decoding unit, and the MAC engine performing convolution with bias.

The ETHOS-NPU accelerates convolution with a dedicated DMA driving AXI interfaces to external memory, a MAC unit with a weight decoder, internal SRAM, and an output unit for feature maps.

Outline the TFLM top level flow: initialize the TensorFlow Lite for Microcontroller runtime and interpreter, allocate memory for nodes and buffers, and dispatch NPU and CPU/DSP kernels via CMSIS NN.

Explore the TensorFlow Lite Micro initialization stage that builds a resolver lookup table of operation registrations, with init, free, prepare, and invoke callbacks for convolution 2D and relu.

Allocate an arena buffer in SRAM for inputs, outputs, and intermediate feature maps, while keeping weights and biases in flash, and define the maximum arena size via the linker script.

Load the TensorFlow Lite model from flash or ROM into the interpreter. Initialize the resolver, allocate the tensor arena for inputs, outputs, and intermediates, then instantiate and invoke the interpreter.

Save the model pointer locally, create an allocator for node and tensor registration, instantiate TensorFlow Lite context to access nodes and tensors, structures to iterate and invoke init, prepare, invoke.

Allocate tensors and buffers in Tensor Arena within SRAM to support inputs, outputs, intermediates, and persistent buffers; parse the input model and map operator nodes to TensorFlow Lite registrations, init/prepare.

Trace the TFLM tensor and node allocation flow, from allocator reset to creating TensorFlow Lite nodes, registration lookups, and operator init, prepare, and invoke callbacks.

Learn how tflm nodes and registration data allocate tensors, register operators via a resolver, and invoke init to prepare buffers and contexts for executing embedded machine learning operations.

Use the local allocator to create a TensorFlow Lite tensors list for all model tensors, embedding type, buffer pointers, quantization, shape, and constant versus dynamic memory to drive operation execution.

After tensor node allocation, preparation, and init, the interpreter iterates over supported operations, extracting invocation callbacks and dispatching execution to ethos-u npu or cmsis macros on cpu or dsp.

Illustrates invoking a tflm activation relu eval via a wrapper that selects npu or fallback to dsp or cpu, using the default kernel software path.

Explains how custom Ethos-U NPU operations are invoked via a standardized interpreter API, preparing workspace and tensor arena, encoding the operation in the input stream, and invoking the NPU driver.

Trace the offline training to on-device execution flow, from TensorFlow Lite optimizations to a flatbuffer running on a microcontroller, with CMSIS-NN dispatching to Cortex-M, Helium DSP, or Ethos-U NPU.

Explore how the CMSIS neural network library targets cortex-m microprocessors and helium dsps with apis. TensorFlow Lite for Microcontrollers parses FlatBuffer data and dispatches operations to cpu, dsp, or npu.

Explore the CMSIS-NN software architecture for embedded ML, covering convolution, pooling, activations, and data type conversion with offline quantization to int8/float8, including lookup table activations to save compute energy.

Compile the cmsis-nn framework for m4 with dsp, enabling armv7em feature flags via cmsis headers to activate dsp-based low-level APIs in the neural network library.

Explore how CMSIS-NN APIs map to neural network layers, showing multiple low-level functions per operation, data-type and pooling scheme variations, and CPU feature considerations.

Explain CMSIS-NN targets for Cortex-M0/M0+, M4/M7, and M55/M85 with DSP and vector extensions; show how feature flags select C, DSP, or Helium implementations in Vela.

Explore the Alif E7 block diagram, highlighting the application processor with dual Cortex-A32 cores on ARM v9-A and two Ethos-U55 NPU subsystems (256 vs 128 MACs) for embedded machine learning.

Compare ETHOS-U55 128 and 256 models, noting 128 vs 256 MAC units for low power and high performance machine learning; 128 suits keyword spotting, 256 suits vision workloads.

Select the Alif E7 development kit with Ethos-U55 NPU for embedded machine learning, featuring a dedicated subsystem, lcd-tft display, and a dedicated camera interface for vision-based, low-power tasks.

Explore the Alif E7 development kit schematic, detailing jumpers, switches including reset, LED indicators, LCD and camera connectors, dual USB ports, quad SPI flash, and J-Link or JTAG debugging.

Configure jumpers on the Alif E7 kit to enable secure enclave output via USB, verify voltage with jumper 5-3-4, and set camera and run-mode jumpers.

Explore the Alif machine learning development kit board with lcd and back camera, usb power; set jumper 15 to connect secure uart to the ftdi chip for debugging.

Power up the alif e7 development kit via the micro USB, verify jumper settings, and use reset and joystick switches to program and run your ml application with secure enclave.

Connect the Alif E7 board to a Windows machine, power it via USB, and program the binary through the UART-to-USB interface, observing the com port in device manager.

Power on the board to boot the ready programmed face detection machine learning application, which detects and classifies faces, and press the joystick button to output inference results.