mirror of
https://github.com/EpicGames/lore
synced 2026-08-19 10:28:47 +00:00
mirror of epic games' lore open source version control
- Rust 83.3%
- Python 12.5%
- C 3.6%
- Shell 0.3%
- HCL 0.2%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
b046bcb2d9
## Motivation Callers that address content by a key belonging to some other system -- an asset id, a build id, a database primary key -- need both keyspaces for every access, and the two are serially dependent: a read is mutable_load then get, a write is put then mutable_store, and the second cannot start until the first answers. The latency is two round trips regardless of how much work is queued behind it, which is what decides whether Lore is usable as a cache for foreign-keyed data. ## Summary Add get_resolved and put_resolved, which resolve a key and act on the content it names within one request. Keys live under KeyType::Resolve, so the type is implied rather than sent. Both operations are available over QUIC and gRPC with the same semantics, and through lore_storage_get_resolved and lore_storage_put_resolved in the C API. A read resolves local-first and falls back to the remote on a miss, caching the mapping it learns; a locally cached mapping that names content this store does not hold defers to the remote rather than reporting a miss. The root fragment arrives with the resolution, so there is no local probe for it -- that is the round trip being saved -- but it is decompressed and verified against the resolved hash exactly as load_fragment does, so a server that resolves a key to the wrong content is caught rather than trusted. Leaves still go through load_fragment, and so are still served from and written back to the local store. Reads take the same streaming option an ordinary read does, delivering one event per leaf fragment so peak memory follows the fragment size rather than the content size. A write stores the content before publishing the key, so a key never names content the store does not hold. Content that fits one fragment publishes from inside the ordinary write pipeline: the upload it was going to perform anyway becomes put_resolved rather than put, so content and mapping travel together and the terminal local write is the only local write, already carrying the durable flag and the local_cache_priority retention decision. A publishing write never joins the in-flight rendezvous -- deduping onto another writer's upload would mean no upload of our own, and the key rides on the upload, so it would silently never ship. Content already durable uploads nothing, so there is no command for its key to ride on and the mapping follows as a mutable_store instead; still one round trip, without re-sending a payload the server holds. A fragment tree cannot fuse: its leaves upload first and the key is published only once the whole tree is durable. Writes report where content landed -- local, remote, or both -- as the intersection across every fragment, so a caller can tell content that reached the remote from content that only reached the local store. A remote leg that fails is warned rather than returned: the local write still stands, and the reported placement says so. A publish carrying no content retracts the key. Publishing is last-writer-wins, deliberately: for the caching this exists for either mapping is valid. Folding in a compare-and-swap is not available, because content must be stored before the key names it and a failed swap would strand what it just stored, worst under exactly the contention that would motivate it. ``` Imported-PR: #169 Imported-From: |
||
| .cargo | ||
| .github | ||
| contrib/aws | ||
| docs | ||
| lore | ||
| lore-aws | ||
| lore-base | ||
| lore-capi | ||
| lore-chaos-client | ||
| lore-client | ||
| lore-credential | ||
| lore-error-set | ||
| lore-error-set-macro | ||
| lore-hashicorp | ||
| lore-integration-tests | ||
| lore-io | ||
| lore-macro | ||
| lore-notification | ||
| lore-proto | ||
| lore-revision | ||
| lore-server | ||
| lore-storage | ||
| lore-telemetry | ||
| lore-transport | ||
| notices | ||
| scripts | ||
| vendor/quinn-proto | ||
| .dockerignore | ||
| .gitignore | ||
| .loreignore | ||
| .markdownlint-cli2.jsonc | ||
| .pre-commit-clippy.py | ||
| .pre-commit-config.yaml | ||
| .rustfmt.toml | ||
| .vale.ini | ||
| about.toml | ||
| build-helper.rs | ||
| Cargo.lock | ||
| Cargo.toml | ||
| clippy.toml | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| DCO | ||
| deny.toml | ||
| GOVERNANCE.md | ||
| LICENSE | ||
| MAINTAINERS.md | ||
| mkdocs.yml | ||
| pyproject.toml | ||
| pyrightconfig.json | ||
| README.md | ||
| SECURITY.md | ||
| uv.lock | ||
Lore
Next-generation open source version control
Download Lore · Quickstart · Read the docs · Join the conversation
Table of contents
About Lore
Lore is an open source version control system designed for unprecedented scalability of both data and teams. It is optimized for projects that combine code with large binary assets, including games and entertainment, and caters for the needs of developers and artists alike.
Note
Lore is pre-1.0 and under active development. Interfaces, on-disk formats, and APIs may change between releases.
Get started with Lore
- Quickstart — install Lore and make your first commit by following the quickstart guide.
- Read the docs — delve into Lore's ethos and architecture in the Lore documentation.
- Have questions? — the FAQ covers licensing, supported platforms, production readiness, and how Lore compares to other version control systems.
- See where Lore is headed — the roadmap lays out the big-rock features by time horizon, from scalable locking to an open source desktop client.
- Join the conversation — chat with us and our community on Discord.
Or try it right now — install Lore and start a local server in demo mode:
macOS / Linux
curl -fsSL https://raw.githubusercontent.com/EpicGames/lore/main/scripts/install.sh | bash -s -- --demo
Windows (PowerShell)
$env:LORE_DEMO=1; irm https://raw.githubusercontent.com/EpicGames/lore/main/scripts/install.ps1 | iex
Overview
- Easy setup, on-demand scalability — Get started in local mode in minutes. Then, scale up as far and as fast as you need.
- Fast and efficient processes — Scale without slowdowns, thanks to shared, reusable data and as-needed downloads.
- Free branching — Quickly and easily create, manage, and sync branches to freely experiment, iterate, and release.
- History you can trust — Confidently track and manage revisions with Lore's verifiable tamper-evident source of truth.
- Intuitive interface — Enjoy complete one-to-one access to the full Lore functionality via the CLI.
- Full-surface API — Extend, customize, and integrate Lore via C/C++, C#, Rust, Go, Python, or JavaScript.
Note
Lore is the built-in version control system for UEFN (Unreal Editor for Fortnite), but today's open source tooling can't yet talk to it: the UEFN build uses a proprietary compression format that can't ship with the open source project. We're actively moving UEFN onto an open compression format — the same one this open source project uses — to eliminate the gap between the two.
Lore's architecture
Lore is a centralized, content-addressed version control system that represents repository state as Merkle trees and an immutable revision chain, optimized for binary-first storage, deduplication, and sparse/on-demand data hydration at scale. For the full model—on-disk formats, chunking internals, and the mechanics of the Merkle tree—read the system design doc.
Highlights
- Content-addressed storage — Repository data is stored and referenced by content hash in a Merkle tree, enabling fast comparisons, integrity checks, and reuse across history and branches.
- Immutable revision chain — A revision's hash signature is derived from its revision state, including parent revision hashes and contained data hashes, forming an immutable chain with cryptographic integrity.
- Chunked storage for large files — Files are stored as reusable chunks with indexed lookup, reducing duplication and enabling efficient updates and transfer for large binary assets.
- On-demand hydration and sparse workspaces — Workspaces can stay lightweight by fetching file data only when needed, so you don't have to download everything up front.
- Centralized service with caching — A service-backed architecture uses caching in front of durable storage to scale throughput for large teams and repositories.
- Lightweight branches and fast switching — Branches are lightweight mutable references, so creating and switching branches is low-overhead without duplication of underlying data.
Lore's repositories
Lore spans a family of repositories: the core library, server, and CLI in this repository, plus a software development kit (SDK) for each supported language.
| Repository | Description | Link |
|---|---|---|
| Lore Library, Server & CLI | The core Lore library, the Lore Server, and the Lore CLI. You are here. | View on GitHub |
| JavaScript SDK | The JavaScript binding for the Lore API. | View on GitHub |
| Python SDK | The Python binding for the Lore API. | View on GitHub |
| C# SDK | The C# binding for the Lore API. | View on GitHub |
| Go SDK | The Go binding for the Lore API. | View on GitHub |
Fully open source
We believe a truly open ecosystem is built collectively using open standards. Lore is fully open source under an MIT license, and we invite you to build the version control system of the future in the open. See CONTRIBUTING.md to get involved.
Contributing
Contributions of every kind are welcome — code, documentation, bug reports, and reviews. Start with CONTRIBUTING.md for the development workflow, then read the Code of Conduct and the project governance model. New to the codebase? The good-first-issue label is a good place to start.
License
Lore is released under the MIT License. See LICENSE for the full text. Copyright (c) 2026 Epic Games, Inc.
Contact and community
- Discord — chat with the team and community on Discord.
- GitHub Issues — report bugs and request features through GitHub Issues.