Purpose

URWTF64 is an open, extensible standard designed to convert continuous real-world data streams into compact, deterministic 64-bit tokens.
These tokens form a universal language for AI systems to interact with, analyze, and act upon real-time physical environments.

URWTF64 v0.2.0:

• Enforces strict canonicalization of values

• Includes an explicit DECIMALS field after every numeric VALUE

• Mandates per-entity checksum tokens for data integrity

Core Principles

• Universality: Domain-agnostic, applicable to energy, logistics, manufacturing, transportation, healthcare, and more

• Extensibility: New tokens can be defined as technologies evolve

• Hierarchy: Organized into Primitive, Aggregate, Composite, and Checksum tokens for simple readings and integrity checks

• Standardization: Canonical string rules ensure consistent 64-bit IDs for identical facts

• Entity Framing: SPECIAL tokens separate and recombine streams from multiple devices or subsystems

Why Tokenization Matters

URWTF64 transforms unstructured data (floats, JSON, logs) into discrete, semantic 64-bit tokens, offering:

• Efficiency: Each fact is 8 bytes, 10–50× smaller than JSON

• Uniformity: Consistent rules across devices prevent schema lock-in

• Interpretability: Tokens represent meaningful states/events for humans and AI

• Real-Time Responsiveness: Compact tokens enable low-latency streaming and action

• Cross-Domain Learning: Unified token space supports model generalization

• Integrity: Mandatory CHECKSUM tokens ensure self-verifiable streams

Token Levels Explained

Primitive Tokens

Raw measurements or direct observations
Format: PRIMITIVE|KEY|VERSION|VALUE|DECIMALS|UNIT

• TYPE: must be PRIMITIVE

• KEY: upper case

• VERSION: X.Y.Z form

• VALUE: numeric, rounded and padded to match DECIMALS

• DECIMALS: integer 0–5

• UNIT: upper case

Example
 PRIMITIVE|FLOW_RATE|0.2.0|12.30|2|L/S

Aggregate Tokens

Summaries over windows or intervals
Format: AGGREGATE|KEY|VERSION|METHOD|INTERVAL|VALUE|DECIMALS|UNIT

• METHOD: upper case ENUM (MEAN, MIN, MAX, SUM, COUNT, STDDEV, VARIANCE, MEDIAN, PERCENTILE_…)

• INTERVAL: ISO 8601 duration, upper case

• VALUE: numeric, rounded and padded to match DECIMALS

• DECIMALS: integer 0–5

• UNIT: upper case

Example
 AGGREGATE|TEMPERATURE|0.2.0|MEAN|PT10M|68.0|1|C

Composite Tokens

Logical conditions, alerts, or multi-sensor rules
Format: COMPOSITE|KEY|VERSION|LABEL

Example
COMPOSITE|ALERT|0.2.0|PRESSURE_LOW

Special Tokens

Structural markers delimiting entities
Format: ENTITY_N-BEGIN, ENTITY_N-END

• ENTITY_N-BEGIN = 0xFF00000000000000 + 2 × (N–1)

• ENTITY_N-END = 0xFF00000000000000 + 2 × (N–1) + 1

Checksum Tokens

Integrity markers per entity, placed before ENTITY_END

Computation

1. Collect all canonical strings within the entity (exclude BEGIN, CHECKSUM, END)

2. Concatenate with newline separators

3. Hash with MurmurHash3_64 (seed 0x5BAE381D5BAE381D)

4. Map into the CHECKSUM range

Note: All fields in canonical strings are separated by [PIPE] (for example PRIMITIVE|FLOW_RATE|0.2.0|12.30|2|L/S).

TokenID Allocation

Notes

• Two SPECIAL tokens per entity (BEGIN, END) → max 36,028,797,018,963,968 entities

• RESERVED is for future extensions

• Total allocation covers 2^64

Canonicalization Rules

Primitive Tokens

Format: PRIMITIVE|KEY|VERSION|VALUE|DECIMALS|UNIT

• Key: upper case

• Type: PRIMITIVE

• Version: X.Y.Z form

• Value: expanded decimal string, rounded and padded to match DECIMALS

• Decimals: integer 0–5

• Unit: upper case

Examples
PRIMITIVE|FLOW_RATE|0.2.0|12.302|3|L/S
PRIMITIVE|PRESSURE|0.2.0|2.10|2|BAR
PRIMITIVE|TEMPERATURE|0.2.0|71.3|1|C

Validation: reject if VALUE’s fractional digits don’t match DECIMALS (use round half even).
Calculation: hash with MurmurHash3_64 and map to PRIMITIVE range.

Aggregate Tokens

Format: AGGREGATE|KEY|VERSION|METHOD|INTERVAL|VALUE|DECIMALS|UNIT

• Method: upper case ENUM (MEAN, MIN, MAX, etc.)

• Interval: ISO 8601 duration

Example  
AGGREGATE|FLOW_RATE|0.2.0|MEAN|PT5M|12.10|2|L/S

Composite Tokens

Format: COMPOSITE|KEY|VERSION|LABEL

• Key: upper case (ALERT, STATE, MODE, RULE)

• Version: X.Y.Z form

• Label: upper case, underscores allowed, no operators/spaces

Example
 COMPOSITE|ALERT|0.2.0|PRESSURE_LOW

Special Tokens

Format: ENTITY_N-BEGIN, ENTITY_N-END
Entities are strictly bounded by BEGIN and END.

Entity Checksums

Mandatory before ENTITY_END

Steps

1. Collect canonical strings (exclude BEGIN, CHECKSUM, END)

2. Concatenate with newline separators

3. Hash with MurmurHash3_64

4. Map to CHECKSUM range

Note: All fields in canonical strings are separated by [PIPE] (for example PRIMITIVE|FLOW_RATE|0.2.0|12.30|2|L/S).

Example Stream (URWTF64 v0.2.0)

TokenId          ; Comment (only TokenIDs are transmitted, comments shown for clarity)
FF00000000000000 ; ENTITY_1-BEGIN (Main Pump)
4ECC7C41AABBCCDD ; [ PRIMITIVE|FLOW_RATE|0.2.0|12.30|2|L/S ]
44BEC44B11223344 ; [ PRIMITIVE|PRESSURE|0.2.0|2.104|3|BAR ]
50314FE5778899AA ; [ PRIMITIVE|TEMPERATURE|0.2.0|71.3|1|C ]
A0D3011AFFEEDDCC ; [ AGGREGATE|FLOW_RATE|0.2.0|MEAN|PT5M|12.10|2|L/S ]
B94A6509AABBCCDD ; [ COMPOSITE|ALERT|0.2.0|PRESSURE_LOW ]
DF9A123456789ABC ; CHECKSUM
FF00000000000001 ; ENTITY_1-END

FF00000000000002 ; ENTITY_2-BEGIN (Backup Pump)
3DF203CA11223344 ; [ PRIMITIVE|FLOW_RATE|0.2.0|0.00|2|L/S ]
67FFC822AABBCCDD ; [ PRIMITIVE|PRESSURE|0.2.0|1.90|2|BAR ]
6E853F1711223344 ; [ PRIMITIVE|VIBRATION_X|0.2.0|0.5500|4|MM/S ]
1F051346FFEEDDCC ; [ PRIMITIVE|VIBRATION_Y|0.2.0|0.6104|4|MM/S ]
50314FE5778899AA ; [ PRIMITIVE|TEMPERATURE|0.2.0|68.2|1|C ]
DFAC901156789ABC ; CHECKSUM
FF00000000000003 ; ENTITY_2-END

FF00000000000004 ; ENTITY_3-BEGIN (Reservoir Tank)
150CF16FAABBCCDD ; [ PRIMITIVE|LEVEL|0.2.0|78.0|1|PERCENT ]
02DE988F11223344 ; [ PRIMITIVE|TEMPERATURE|0.2.0|15.6|1|C ]
ACAC70E7FFEEDDCC ; [ AGGREGATE|LEVEL|0.2.0|MEAN|PT1H|77.50|2|PERCENT ]
B939D283AABBCCDD ; [ COMPOSITE|ALERT|0.2.0|HIGH_LEVEL ]
DFBC77EE56789ABC ; CHECKSUM
FF00000000000005 ; ENTITY_3-END

FF00000000000000 ; ENTITY_1-BEGIN (Main Pump)
67BEC83C11223344 ; [ PRIMITIVE|PRESSURE|0.2.0|2.201|3|BAR ]
DFE823456789ABCD ; CHECKSUM
FF00000000000001 ; ENTITY_1-END

Note on Collisions

URWTF64 mitigates collisions via:

• Checksums: detect mismatches per entity

• Registry: pregenerated canonical strings with fixed definitions

• Universal Catalog: maps TokenIDs to canonical strings

• Local Overlays: vendor/site-specific strings validated by checksums

AI and LLM Integration: LLM_READY_LOOKUP_TABLE

Goal

Enable URWTF64 streams to be used in LLM training by compressing the token space into a manageable vocabulary.
The aim is to minimize the number of distinct tokens while still preserving semantic meaning, sometimes by grouping nearby values so the model sees coherent categories rather than thousands of tiny distinctions.
This makes the token-space comprehensive yet efficient for training.

Method

1. Vocabulary Build

   • Scan URWTF64 streams

   • Keep ENTITY_BEGIN and PRIMITIVE / AGGREGATE / COMPOSITE tokens

   • Exclude CHECKSUM and ENTITY_END

   • Group tokens with high-precision values into shared buckets (e.g., rounding or clustering) to reduce sparsity and strengthen learning

   • Assign dense integer indices (0..V–1) to individual or grouped tokens

2. Materialize Sequences

   • Re-scan the stream

   • Replace TokenIDs with their assigned index (grouped if applicable)

   • Output compact integer sequences ready for training

Benefits

• Compression: Drastically reduces vocabulary size by grouping similar tokens

• Stronger Training Signal: Nearby numeric values collapse into shared tokens, letting the model generalize better

• Small Vocabulary: Tens of thousands to a few million tokens instead of 2^64

• Efficient Training: Smaller embedding matrices, reduced memory, faster convergence

• Collision-Free: Checksums still ensure that canonical streams remain verifiable

• Context Preservation: ENTITY_BEGIN markers retain device/session structure

Extending the Framework

• New Tokens: hash new canonical strings into correct ranges

• Versioning: increment version when rules, seeds, or hashing change

• Community: encourage contributions for broad coverage

Implementation Guidelines

• Data Sources: IoT devices, sensors, controllers

• Processing: edge canonicalization and hashing

• Storage: time-series DBs or message buses

• Integration: feeds anomaly detection, LLMs, automation

Next Steps

• Pilot in energy and manufacturing

• Build open canonical token repository

• Collaborate with standards bodies for adoption

Call to Action

URWTF64 v0.2.0 bridges physical reality and AI with a shared, efficient, deterministic language.
Engineers, developers, and researchers are invited to adopt and extend URWTF64.