Saving PyTorch tensors through the adapter
Notebindings/python/py_src/safetensors/torch.py
`save_file` accepts a mapping of names to dense contiguous tensors, optional string metadata, and a destination path. It flattens tensor storage into descriptors and delegates file serialization to the binding, retaining temporary references for the duration of the write.
The function documents that callers must avoid mutating tensor storage concurrently with saving. This adapter is responsible for framework storage constraints; the Rust core owns the on-disk layout and metadata checks.
Evidence: [bindings/python/py_src/safetensors/torch.py](lumvise://element/filesystem%3Acc55918e7125afba%3Abindings%2Fpython%2Fpy_src%2Fsafetensors%2Ftorch.py%3Afile%3Atorch.py%3A), [safetensors/src/tensor.rs](lumvise://element/filesystem%3Acc55918e7125afba%3Asafetensors%2Fsrc%2Ftensor.rs%3Afile%3Atensor.rs%3A).
Shared tensor storage needs model-aware handling
Notebindings/python/py_src/safetensors/torch.py
`save_model` obtains the model state dictionary, detects shared tensor storage, chooses a complete storage representative, records dropped names in metadata, and optionally makes tensors contiguous before saving. It refuses a shared group when no candidate covers the complete storage.
This is needed because names in a PyTorch state dictionary can alias storage, while independent tensor entries in the file do not encode the entire Python alias graph. The metadata can explain dropped names but is not enough by itself to reconstruct every sharing relationship.
Evidence: [bindings/python/py_src/safetensors/torch.py](lumvise://element/filesystem%3Acc55918e7125afba%3Abindings%2Fpython%2Fpy_src%2Fsafetensors%2Ftorch.py%3Afile%3Atorch.py%3A).
Validate header, tensor extents, and buffer coverage
Notesafetensors/src/tensor.rs
`read_metadata` reads the little-endian header size, checks bounds, decodes UTF-8/JSON, converts metadata, and calls `validate`. It then requires the declared tensor data plus header to consume the complete input buffer.
`validate` checks contiguous ordered offsets, nonnegative extents, checked shape multiplication, dtype bit size, byte alignment, and exact byte lengths. These checks prevent malformed metadata from producing out-of-range tensor views.
The indexed Rust file has a parser recovery warning; this explanation was checked against the actual source, and the selected function's MCP snippet matched the worktree.
Evidence: [safetensors/src/tensor.rs](lumvise://element/filesystem%3Acc55918e7125afba%3Asafetensors%2Fsrc%2Ftensor.rs%3Afile%3Atensor.rs%3A).
Loading tensors with an explicit device and backend
Definitionbindings/python/py_src/safetensors/torch.py
`load_file` opens the archive through `safe_open` with framework `pt`, the requested device, and a storage backend, then returns its tensors as a name-to-tensor mapping. In this local snapshot the default backend is `mmap`; `pread` is also documented.
The convenience function loads the mapping. The lower-level `safe_open` interface provides more direct control over tensor access. These behaviors describe the locally indexed checkout, whose upstream commit metadata is absent.
Evidence: [bindings/python/py_src/safetensors/torch.py](lumvise://element/filesystem%3Acc55918e7125afba%3Abindings%2Fpython%2Fpy_src%2Fsafetensors%2Ftorch.py%3Afile%3Atorch.py%3A), [bindings/python/src/lib.rs](lumvise://element/filesystem%3Acc55918e7125afba%3Abindings%2Fpython%2Fsrc%2Flib.rs%3Afile%3Alib.rs%3A).
Tensor archive layout and borrowed views
Definitionsafetensors/src/tensor.rs
A safetensors file begins with an eight-byte little-endian header length, then a UTF-8 JSON header, then tensor bytes. Each tensor entry records dtype, shape, and an offset interval relative to the data section. Optional `__metadata__` values are strings.
`SafeTensors` owns lookup metadata while borrowing the data buffer. This enables views into validated byte ranges. Zero-copy behavior at a framework boundary still depends on storage backend, device, and conversion needs.
This is the upstream tensor format. The enclosing Lumvise demo graph uses the separate `.pz` format.
Evidence: [README.md](lumvise://element/filesystem%3Acc55918e7125afba%3AREADME.md%3Afile%3AREADME.md%3A), [safetensors/src/tensor.rs](lumvise://element/filesystem%3Acc55918e7125afba%3Asafetensors%2Fsrc%2Ftensor.rs%3Afile%3Atensor.rs%3A).
Functional meaning: py.typed
Summarybindings/python/py_src/safetensors/py.typed
## Job
marker_file
## Source Interface
PEP 561 type marker (py.typed, zero-length span)
## Receives
- None declared by immediate child artifacts.
## Outcome
Empty py.typed marker file (PEP 561); its presence marks the safetensors Python package as inline-typed; no executable content in span.
## Notable Effects
- None declared by immediate child artifacts.
Start here: safetensors knowledge graph
GuideREADME.md
# safetensors source tour
This demo combines the complete published semantic index with selected explanations attached to real files, folders, classes, and functions. Start with architecture, then follow the core concepts:
1. [Safetensors architecture: format core and framework adapters](lumvise://artifact/popular-demo-20260928%3Asafetensors%3Aarchitecture)
2. [Tensor archive layout and borrowed views](lumvise://artifact/popular-demo-20260928%3Asafetensors%3Aformat)
3. [Validate header, tensor extents, and buffer coverage](lumvise://artifact/popular-demo-20260928%3Asafetensors%3Avalidation)
4. [Saving PyTorch tensors through the adapter](lumvise://artifact/popular-demo-20260928%3Asafetensors%3Asave)
5. [Loading tensors with an explicit device and backend](lumvise://artifact/popular-demo-20260928%3Asafetensors%3Aload)
6. [Shared tensor storage needs model-aware handling](lumvise://artifact/popular-demo-20260928%3Asafetensors%3Asharing)
7. [Source snapshot, index coverage, and validation scope](lumvise://artifact/popular-demo-20260928%3Asafetensors%3Aprovenance)
Select an artifact to inspect its owning semantic element. Evidence links point to indexed source. The source snapshot and coverage report records the exact scope and parser limitations.
Safetensors architecture: format core and framework adapters
Reportsafetensors
# Safetensors architecture
The Rust crate owns tensor metadata, byte layout, validation, serialization, and borrowed tensor views. Python bindings expose this core, while framework modules adapt NumPy, PyTorch, TensorFlow, Flax, Paddle, and MLX tensor objects.
```text
framework tensors → Python adapter → Rust serialization → file
file → validated metadata + tensor bytes → framework tensors
```
`SafeTensors` references a shared byte buffer instead of owning a copy of every tensor. Framework adapters translate dtype, shape, storage, and device behavior. The format stores tensors and text metadata rather than executable Python objects; model architecture still belongs to the consuming framework.
Read the format and validation artifacts before the PyTorch adapter notes.
Evidence: [safetensors/src/lib.rs](lumvise://element/filesystem%3Acc55918e7125afba%3Asafetensors%2Fsrc%2Flib.rs%3Afile%3Alib.rs%3A), [safetensors/src/tensor.rs](lumvise://element/filesystem%3Acc55918e7125afba%3Asafetensors%2Fsrc%2Ftensor.rs%3Afile%3Atensor.rs%3A), [bindings/python/src/lib.rs](lumvise://element/filesystem%3Acc55918e7125afba%3Abindings%2Fpython%2Fsrc%2Flib.rs%3Afile%3Alib.rs%3A), [bindings/python/py_src/safetensors/torch.py](lumvise://element/filesystem%3Acc55918e7125afba%3Abindings%2Fpython%2Fpy_src%2Fsafetensors%2Ftorch.py%3Afile%3Atorch.py%3A).
Source snapshot, index coverage, and validation scope
ReportREADME.md
# Export provenance
Upstream: [safetensors/safetensors](https://github.com/safetensors/safetensors).
This graph was generated on 2026-09-28 from the existing local source folder. The folder has no Git metadata, so an exact upstream commit is unknown; no branch or commit is guessed. It was not updated from upstream during export.
Source snapshot fingerprint: `f28c496fe189bde9b7424a53583b1effeadd25985d72f867864060f406ed1b4a` (SHA-256 over sorted relative paths, NUL separators, and raw file SHA-256 digests; excludes Git/runtime/generated cache directories and symlinks). Regular source files: 115. Indexed semantic elements: 1714. File/text parser records: 115 (49 plain_text, 65 parsed, 1 unsupported); images have separate semantic kinds.
Coverage details:
- `bindings/python/py_src/safetensors/py.typed`: unsupported.
- `safetensors/src/tensor.rs`: parsed; parser recovered from syntax errors.
The scanner skipped `safetensors/LICENSE` and `safetensors/README.md` symlinks; root LICENSE and README were indexed.
Static extraction is best effort. Unresolved dynamic calls are not evidence that dependencies are absent. Knowledge explanations were checked against selected local source; upstream test suites, notebooks, model inference, and model downloads were not run. The task validates index/export contents and readability.