A result must survive the person and process that produced it
Imagine our nitrogen calculation has finished. We have an energy, an active-space Hamiltonian, a trial wavefunction, circuits, and a conditional resource estimate. A colleague wants to compare a better orbital-selection method next month. Another colleague wants to run the same quantum workload against a different hardware model. A third needs to understand a surprising discrepancy after a software update.
Sending them one energy or a circuit screenshot is not enough. They need scientific artifacts with conventions, identities, and a history. This workstream is where reproducibility becomes an engineering task rather than an aspiration.
Serialization converts an in-memory object into a representation that can be saved or transmitted. Deserialization reconstructs an object from that representation. A schema defines the fields, shapes, types, and conventions of the saved representation. Provenance is the record of how an artifact was produced: its inputs, choices, software, and relationship to other artifacts.
These concepts overlap, but they solve different problems. A file can deserialize perfectly while its scientific provenance is missing. A beautifully documented experiment can still be unusable if the only saved object cannot be loaded. We need both.
The serialization guide and common data interface describe the shared mechanisms. Individual data classes determine what information is actually preserved.
1. Choose a format according to what must survive
QDK Chemistry uses several formats for different purposes:
| Format | Useful role | What not to assume |
|---|---|---|
| JSON | Inspectable structured text containing fields, lists, numbers, and nested records | Human-readable does not mean small, convention-free, or universally interchangeable. |
| HDF5 | Hierarchical numerical storage with groups, arrays, and attributes | The container format does not explain the scientific meaning of every array. |
| XYZ | Simple molecular geometry interchange | Charge, spin, full precision, custom isotope masses, and all model settings are not preserved by the geometry alone. |
| FCIDUMP | Exchange of electronic Hamiltonian integrals and electron-sector information | It is not a complete archive of an arbitrary QDK workflow or unrestricted Hamiltonian. |
| Cube | A spatially sampled orbital or density for visualization | A sampled picture is not the full orbital basis, wavefunction, or Hamiltonian. |
JSON is convenient when we want to inspect a small artifact in a text editor. HDF5 can store multidimensional arrays without spelling every number as text. Its groups act somewhat like folders, its datasets hold arrays, and its attributes hold associated metadata. Those are capabilities of HDF5; using it does not automatically add compression, scientific validation, or a complete experiment record.
The package uses filenames such as nitrogen.structure.json, selected.orbitals.h5, and reference.wavefunction.h5. The type in the middle matters: the loader checks a name.type.extension convention. Giving a wavefunction file a structure suffix does not convert its contents into a structure. The Python base performs these checks before opening files.
An instructional round trip looks like this:
from qdk_chemistry.data import Wavefunction
reference.to_hdf5_file("reference.wavefunction.h5")
restored = Wavefunction.from_hdf5_file("reference.wavefunction.h5")
This is a usage example, not an operation performed during course preparation. These methods write files; a read-only inspection should not call them merely to learn what they do.
A round trip means saving and loading again. Success should be assessed against the relevant scientific properties: orbital identities, dimensions, coefficients, spin blocks, normalization, and energy constants. Merely obtaining an object of the right class is a weaker check.
2. Numerical arrays have conventions that text alone cannot convey
Suppose one program saves an array of sixteen numbers. Does it describe a four-by-four matrix, a two-by-two-by-two-by-two tensor, or sixteen unrelated coefficients? Which index changes fastest? Are the values real or complex? Which electron-spin block does the array belong to?
The array serialization helpers illustrate explicit answers. Real arrays are stored with both their values and shape. Complex arrays can be represented by real and imaginary parts, a shape, and a numerical dtype. Dtype specifies the numerical representation, such as a particular floating-point precision.
A complex amplitude is not interchangeable with its real part. Losing its imaginary component can change interference and therefore the state. Likewise, rearranging array storage without rearranging index meaning can silently change a Hamiltonian.
The molecular data model adds further structure. A restricted calculation can share alpha and beta orbital data. An unrestricted calculation may have distinct blocks. Active and inactive orbital-index sets connect an integral tensor to the relevant orbital basis. A sparse container records selected entries; a factorized container records factors from which entries can be reconstructed. Those are not equivalent file layouts, even when they represent the same operator mathematically.
One concrete historical issue makes this less abstract. Older HDF5 matrices were written from Eigen column-major storage into datasets whose dimensions were recorded as rows and columns. A generic reader exposing the raw bytes in another order would misinterpret the logical matrix. The migration helper read_matrix deliberately reshapes and transposes that legacy layout. This is not a reason to transpose every HDF5 matrix; it is a conversion for a particular old writer contract.
The lesson is that file extensions do not encode all conventions. A compatible writer and loader do.
3. A scientific object and a complete experiment are different artifacts
A saved Structure retains structural data in its schema, but charge and target multiplicity still belong to the calculation inputs. A Hamiltonian carries its core-energy contribution, but the person comparing results must know whether a later qubit operator or energy decoder already included that contribution. A Wavefunction uses a particular container, whose available information may be determinant coefficients, excitation amplitudes, or selected reduced-density information.
Serialization cannot manufacture missing information. An amplitude container does not become an explicit determinant expansion simply because it has been saved and reloaded. A quantum result object that does not retain every original measurement histogram cannot reconstruct those histograms later. A live Q# callable inside a circuit is process-bound behavior, not simply a JSON number that can be restored by a generic loader. The circuit representation and metadata that survive must be inspected, as discussed in Lesson 17.
For our nitrogen story, a reproducible record should connect the structure, basis, charge and spin target, SCF settings and outcome, selected orbital space, Hamiltonian, correlated reference, chosen trial support and coefficients, qubit encoding, circuit construction, measurement or phase-decoding choices, and resource-model assumptions. The schema of one object is only one part of that record.
An artifact graph is a useful way to think about this: each node is a saved object or result, and each edge means that a later result used an earlier artifact. The goal is not to save every temporary variable, but to preserve the decisions and data needed to repeat, compare, and explain the computation.
4. Schema versions are not package versions
Every data class can carry its own serialization version. That version describes the layout of that type, not the entire package release. Two classes in the same package can have different schema versions, and a package release can leave one schema unchanged while changing another.
The shared version validator accepts an exact match. Otherwise it parses major, minor, and patch numbers: the major and minor components must match, while patch differences are allowed. Therefore the broad documentation phrase that only the current version is accepted should not be read as a byte-for-byte patch-version requirement.
A major or minor mismatch raises an error and directs users toward the migration utility. Silently reading an old layout as a new one would be much more dangerous than refusing to load it: the resulting arrays might have the right shape but the wrong meaning.
A migration transforms an old schema into the new schema through explicit, version-specific steps. It is not a new chemistry calculation and does not improve the physical accuracy of the saved model.
The supported interface is:
from qdk_chemistry import migrate
migrate.convert_file(
"old.wavefunction.json",
"new.wavefunction.h5",
)
The equivalent command is python -m qdk_chemistry.migrate OLD_FILE NEW_FILE. Input and output formats can differ. This is again instructional syntax, not a request to modify files in the source checkout.
The converter detects the data type from the filename and the format from the extension. It refuses to migrate in place, so the original remains available. A migration already at the current schema, a source with no registered conversion step, ambiguous type information, or unsupported data produces MigrationError rather than a guessed conversion.
After transforming the document, it asks the current data-class loader to reconstruct the object. This final validation prevents a migration script from declaring success merely because it created a dictionary.
5. What the migrations actually change
The migration guide describes several concrete transitions:
- Orbital coefficients and energies move into symmetry-blocked storage, retaining the associated basis and index information.
- Hamiltonian integral containers move to their symmetry-blocked layouts.
- Older single-determinant, complete-active-space, and selected-CI wavefunctions move into the state-vector container family.
- Older MP2 and coupled-cluster representations move into the amplitude container.
- An
Ansatzmigrates its embedded Hamiltonian and wavefunction through their respective chains. QpeResultretains its result information while removing an obsolete evolution-time field from the old layout.
Some transformations necessarily change the storage representation without claiming new scientific knowledge. An early Cholesky container saved the full four-center integrals but not the molecular-orbital Cholesky vectors needed by the current factorized representation. The converter therefore preserves those integrals as a canonical four-center Hamiltonian. It does not fabricate missing factors. A later legacy layout that actually stored the required molecular-orbital vectors can retain a Cholesky representation.
The converter also explicitly lacks support for old TimeEvolutionUnitary files that did not save the scale required by the current representation. Regeneration is necessary; a guessed scale would change the interpreted operation. Complex coefficients and complex active reduced-density data are not generally handled by these migration paths and are rejected. That limitation is specific to migration, not a claim that every current data class lacks complex-number support.
The migration examples and fixtures show how exact old layouts are reconstructed and compared with expected new fields. They demonstrate the intended contract in source; no migrations or tests were executed for this lesson.
6. Content hashes identify cached work, not scientific truth
A hash maps a structured input to a compact identifier. If an expensive calculation is requested again with the same identified inputs, a cache can use that identifier to locate a previous result.
Native DataClass.content_hash() defaults to sixteen hexadecimal characters. The native contract explicitly treats these as cache keys within a compatible build, not stable archival identifiers across releases. Defining input data participates; lazily computed caches are excluded according to the class contract.
The Python hashing helpers encode types, lengths, array shapes, dtype, and values. Arrays are made contiguous and represented with a chosen byte order. Strings can be byte-exact or explicitly normalized where the caller uses the normalized helper. It is wrong to assume that every textual spelling or mathematically equivalent representation will hash identically.
The algorithm-run hash incorporates the algorithm type and implementation, sorted setting names and values, positional inputs, and sorted keyword inputs. Positional and keyword passing remain structurally distinct. Two call forms that ultimately request the same scientific operation need not receive the same identifier.
A short hash also has a finite collision probability: different inputs can in principle share it. More importantly, the hash does not automatically include every external cause of numerical change. Dependency version, compiled numerical kernels, machine characteristics, and random behavior require explicit provenance unless the particular identity mechanism includes them. A cached stochastic computation with a fixed configuration is one stored realization, not a new independent sample every time it is returned.
The practical rule is to record the explicit scientific inputs and software environment alongside hashes. Hashes help us find artifacts; they do not replace the explanation of what those artifacts mean.
7. A transport manifest connects metadata to numerical files
Passing a calculation between processes needs more than saving one data class. The file serializer writes a manifest containing typed references to separate artifacts.
A manifest is an index: it states which values exist, how to interpret them, and which files contain larger payloads. Ordinary primitive values such as numbers, booleans, strings, and None can live directly in JSON. QDK data objects use HDF5 artifacts named by content identity. Supported NumPy arrays use separate .npy files, with loading that does not depend on Python pickle execution.
Supplementary · Conceptual diagram
Supported lists and tuples can nest these values. Arbitrary Python callbacks and object arrays do not become portable automatically.
Model and source. The pinned serializer’s supported categories described here: primitive JSON values, typed QDK HDF5 artifacts, and supported non-object NumPy arrays. This is not a universal Python-object serialization promise.
How to read this. File identity and format are only part of reproducibility. Keep scientific conventions and production history with the artifact relationships.
Return to this explanationThe serializer supports nested lists and tuples of supported values. Arbitrary dictionaries are not a general serializable value type in this protocol; dictionaries are used by the protocol itself. A nested AlgorithmRef is a specially supported record carrying algorithm identity and nested settings. Object arrays and structured NumPy dtypes are rejected on the inspected path.
These details explain why an arbitrary Python callback cannot be assumed to run remotely merely because an in-process function accepts it. A value must have a defined wire representation and a compatible reconstruction mechanism at the destination.
Wire representation means the format used across that boundary, even if the boundary is only a local subprocess rather than a network. The data registry associates stable type names with loaders. Plugins that define new data classes must register their loaders in both relevant environments.
The transport code writes staged artifacts before replacing the live manifest, and uses temporary files plus atomic replacement for key writes. An atomic replacement makes the name switch occur as one filesystem operation rather than exposing half-written JSON. Manifest paths are resolved within the designated directory and checked against traversal outside it. Those are concrete transport safeguards, not a complete authentication or trust system for arbitrary external datasets.
8. FCIDUMP is a useful exchange view with narrower scope
The Hamiltonian exposes an FCIDUMP writer. FCIDUMP is a conventional text representation used to share electronic-integral problems with many-electron solvers.
The header records quantities such as orbital count, electron count, and the alpha-minus-beta spin projection. Integral entries use one-based orbital indices, unlike the zero-based indices used throughout much of the Python API. The inspected writer uses the trivial C1 orbital-symmetry labels; it is not exporting a discovered molecular point-group classification.
This writer rejects unrestricted Hamiltonians. It obtains two-electron integrals through the container accessor, so exporting a compact factorization can trigger reconstruction of a large four-index tensor. A compact in-memory object does not guarantee compact or cheap interchange output.
FCIDUMP communicates an integral problem. It does not preserve every orbital picture, arbitrary quantum mapping table, state-preparation callable, shot histogram, or hardware-estimation parameter. The presence of an export method also should not be read as a promise that every FCIDUMP dialect can be imported into every high-level class.
Use an exchange format when connecting a compatible solver. Keep the richer native artifact and provenance when reproducibility requires more than the exchange view.
9. Cube data turns functions into pictures without becoming the original function
To visualize an orbital, we evaluate it at points on a three-dimensional grid. A cube file records that sampled scalar field along with grid and structural information.
The CubeGrid and CubeGenerator API defines an axis-aligned grid by origin, spacing, and point counts. Lengths are in bohr. A bounding grid pads the molecular extent by a margin. The first and last points lie on the padded box; a one-point axis has zero spacing. Invalid zero dimensions and excessive point counts are explicitly rejected.
CubeGenerator.orbital evaluates one orbital from its basis coefficients. density evaluates the supplied density matrix as a quadratic combination of basis functions. It does not automatically add spin blocks or multiply a restricted density by two. Two density matrices can have identical shape yet represent total density and half that total. Shape validation alone cannot detect the scientific distinction.
The evaluator uses the basis ordering and normalization contract; coefficients copied from another package may require permutation. With an ECP, the cube describes the explicitly represented electron contribution. Missing replaced-core density is a feature of that model, not evidence that the plotting engine lost electrons.
The Python convenience function generate_cubefiles_from_orbitals offers native and optional PySCF evaluation. A subtle operational fact is that output_folder=None still uses temporary intermediate files before returning text, despite prose suggesting otherwise. The lower-level generator can evaluate without writing by using an empty output filename. These distinctions matter in read-only, restricted, or remote environments.
Finally, a visually smooth orbital is not a guarantee of numerical accuracy. Grid spacing, extent, selected isosurface value, and sign coloring determine what we see. Isosurface means a surface joining points with the same field value. It is a visualization of sampled information, not the full analytical basis representation or a literal boundary of an electron.
10. The handoff we want to be able to reproduce
A useful saved calculation identifies the scientific model, the intermediate data, the algorithm recipe, the actual result, and the conditions under which it was produced. We preserve original inputs before migration, use a compatible loader rather than guessing array conventions, and distinguish archival records from disposable caches.
For nitrogen, the geometry must still say 1.85 Å rather than silently becoming the separate notebook example. The active Hamiltonian and core energy must still describe the same chosen space. A restored trial state must use the same orbitals and occupation convention. A reported energy must still be traceable to its measurement or phase decoder. A resource estimate must still identify the circuit and hardware assumptions it counted.
Saving those relationships lets another team improve one stage and measure its consequences. The next lesson uses the same serialization and identity machinery to move work between processes and machines without losing the meaning of the computation.