# ByteBin: Codebase and User Guide

ByteBin is a developer workspace for organizing code projects, exploring GitHub repositories, asking AI questions about code, finding relevant commits, and collaborating with other ByteBin users. **Tlux** is ByteBin’s task-oriented assistant.

This brief explains the product in one pass: what a user does, what the code does behind the scenes, and where the data goes. The full handbook is available in the app at `/docs`; the interactive service diagram is at `/docs/bytebin-architecture.html`.

## The two ways to work with code

ByteBin has two distinct repository workflows:

| Workspace | What it contains | How code gets there |
| --- | --- | --- |
| **Library** | ByteBin-managed projects and files, including main projects and sub-projects | Create a project in the web app or with Tlux; upload files or use the `byte` CLI |
| **Git Control** | GitHub repositories authorized through ByteBin’s GitHub App | Connect GitHub, select an installed repository, then browse, edit, review, and push changes |

The `byte` CLI uploads to a ByteBin Library project. It does **not** push to GitHub. GitHub commits are made through Git Control after the user reviews and pushes the proposed changes.

## How a user works in ByteBin

1. **Create an account and sign in.** The Next.js app handles the web experience and authenticates requests before they access user data.
2. **Choose a workspace.** Use the Library for a ByteBin project, or connect the GitHub App and select a GitHub repository in Git Control.
3. **Add or explore code.** Library code can come from the UI or CLI. Git Control reads files from the selected GitHub repository.
4. **Ask a question or find a change.** Index a repository to ask code-aware questions, or use Commit Finder to ask where a feature or change appeared in commit history.
5. **Review before changing remote code.** Git Control presents AI-proposed edits for review. Applying the edit and pushing it to GitHub are separate actions.
6. **Continue with Tlux or collaborators.** Tlux can perform supported account, project, and connection tasks. Connections and repository visibility are managed from the dashboard.
7. **Save a conversation only when wanted.** The explicit Save history action queues it for persistence. If saving is off, ByteBin does not publish a history-save event.

## How requests move through the codebase

```mermaid
flowchart LR
  User[Developer]
  Web[Next.js client and API]
  CLI[byte CLI]
  Gateway[Express API gateway]
  AI[AI and code service]
  Files[File service]
  PG[(PostgreSQL)]
  Mongo[(MongoDB)]
  Kafka[(Kafka chat.history)]
  Worker[Chat history worker]
  GitHub[GitHub API]
  Models[Groq and Gemini]
  Email[Email service]
  Resend[Resend]

  User --> Web
  User --> CLI
  CLI -->|login and token refresh| Web
  CLI -->|Library push or update| Gateway
  Web -->|repository, RAG, and commit APIs| Gateway
  Web -->|account and project data| PG
  Web -->|authorized repository calls| GitHub
  Web -->|Tlux assistant request| Models
  Gateway --> AI
  Gateway --> Files
  AI -->|repository file reads| Mongo
  AI -->|embeddings and retrieval| PG
  AI --> Models
  Files -->|Library file documents| Mongo
  Web -.->|only on explicit save or delete| Kafka
  Kafka --> Worker
  Worker -->|persist saved conversations| PG
  Web --> Email
  Email --> Resend
```

### Main service responsibilities

- **`client/` — Next.js web app.** Serves the homepage, dashboard, Library, Git Control screens, Tlux UI, authentication, and protected API routes. It owns most account-facing PostgreSQL access and calls external integrations from the server.
- **`api-gateway/` — Express gateway.** Routes Library file operations, code push/update, repository Q&A, and commit search to their backend services.
- **`file-service/` — Library file API.** Reads and writes ByteBin project file documents in MongoDB. Internal requests are protected with a service key.
- **`llms-service/` — AI and repository processing.** Handles code push/update flows, repository indexing and Q&A, and commit retrieval and explanation. It calls configured AI providers when needed.
- **`chat-worker/` — background history consumer.** Reads Kafka events, validates them, retries transient database errors, and persists saved conversations in PostgreSQL.
- **`email-service/` — welcome email delivery.** Accepts internal, authenticated requests from the web app and sends welcome email through Resend when configured.
- **`byte/` — the `byte` CLI package.** Authenticates the developer, stages selected local files, and sends Library push/update requests through the gateway.

Default local service ports are client `3000`, gateway `8008`, AI service `8009`, file service `8011`, and email service `8012`. Environment variables can change these values.

## AI, indexing, and commit search

### Repository Q&A

Repository Q&A uses retrieval augmented generation (RAG):

1. Indexing splits text files into chunks of 1,200 characters with 200 characters of overlap. Binary files containing NUL bytes are skipped.
2. Gemini generates a 768-dimensional embedding for each chunk. ByteBin stores the text, path, and embedding as PostgreSQL JSONB records.
3. For a question, ByteBin embeds the question, calculates cosine similarity against that project’s chunks, and sends the five highest-ranked passages by default to the chat model as context.
4. The answer can use the recent conversation messages supplied by the chat UI. Re-index after code changes when you need answers to reflect the updated files.

### Commit Finder

Commit Finder loads commits from the authorized GitHub repository. It combines text matching with semantic vector ranking when embeddings are available; if semantic retrieval is unavailable, it can fall back to lexical ranking. Commit embeddings are cached in PostgreSQL. The configured chat model explains the retrieved commits, and the answer includes commit references for verification.

### Tlux

Tlux is an authenticated tool-using assistant. Its current tools can find a ByteBin account by merchant code, send a connection request, create a Library project or sub-project, and delete one. It should ask before sending a request and before creating or deleting a project. Tlux does not run arbitrary terminal commands or push code to GitHub.

## Conversation memory and opt-in history

The model does not acquire permanent memory by itself. ByteBin provides context with each request:

- **Working context:** recent messages are sent with the current question. For long Tlux conversations, older turns are condensed into a summary that preserves goals, decisions, and unresolved items.
- **Saved history:** the user explicitly saves a conversation. The Next.js API publishes a full conversation snapshot to Kafka topic `chat.history`; the worker validates it and writes it to PostgreSQL. The API only reports that the save was queued after Kafka acknowledges the event.
- **No save:** with history saving off, ByteBin does not publish a history-save event or write that conversation to the history tables.
- **Continue later:** the signed-in user can list and reload saved Tlux conversations. Messages and summaries are stored in PostgreSQL and expire seven days after the last save. Deleting a saved conversation is also queued through Kafka.

This is conversation context and saved application data, not model training and not unlimited long-term memory.

## Data ownership

| Store | What ByteBin currently stores there |
| --- | --- |
| **PostgreSQL** | Accounts and tokens, Library project metadata, GitHub installations and repository metadata, connections and visibility, notifications, saved conversations, and code/commit embeddings |
| **MongoDB** | ByteBin Library repository documents and their file contents |
| **Kafka** | A stream of explicit conversation save/delete events while they are being delivered to the worker |
| **Upstash Redis** | Optional best-effort caches; caching is disabled if it is not configured |

PostgreSQL owns account and project identity. The MongoDB document stores the Library file data and carries the matching project ID. Search embeddings are JSONB in PostgreSQL; the current code computes cosine similarity in the AI service rather than requiring a separate vector database.

## Use the `byte` CLI

From this repository, link the local CLI package once:

```sh
cd /path/to/ByteBin/byte
npm install
npm link
```

Then, from the local project folder you want to upload:

```sh
byte login
byte whoami
byte add .
byte push "Bytebin/my-project"
```

Common commands:

| Command | Purpose |
| --- | --- |
| `byte login` | Sign in and save a CLI session |
| `byte add .` | Stage readable files under the current directory |
| `byte add src/file.ts` | Stage selected paths |
| `byte push CODE-ID` | Push staged files into a ByteBin Library project |
| `byte update CODE-ID` | Update an existing Library project or sub-project |
| `byte whoami` | Show the signed-in account and session status |
| `byte logout` | Revoke the CLI session and clear local credentials |

The CLI stores its staging index under `.byte/` and creates a `.gitignore` for it. It skips common generated or hidden directories, environment files, private keys, and files larger than 10 MB. Review staged files before pushing, especially when uploading a real project.

## Run it locally

The root `npm run dev` starts local Kafka and the observability containers, then starts the client, gateway, AI service, file service, chat worker, and email service. It does not provision PostgreSQL, MongoDB, GitHub App credentials, or AI/email provider credentials. Configure the needed values from the `.env.example` files before using those features. Docker is needed for the local Kafka and monitoring containers.

Optional monitoring uses Prometheus for metrics, Loki for logs, Alloy to collect logs, Grafana for dashboards, and Kafka Exporter for broker metrics. The root scripts include `npm run monitoring`, `npm run monitoring:down`, `npm run kafka`, and `npm run kafka:down`.

## Codebase map

```text
client/src/app/             Next.js pages and API routes
client/src/features/        Product UI: auth, dashboard, Git, assistant, commits, docs
client/src/server/          Auth, PostgreSQL repositories, GitHub, Kafka, service clients
api-gateway/src/             Express API routing and service proxies
llms-service/src/            RAG, code push/update, commit retrieval, model adapters
file-service/src/             MongoDB-backed Library file operations
chat-worker/src/              Kafka consumer and history persistence
email-service/src/             Internal welcome-email API
byte/bin/                      CLI entry point
monitoring/                    Prometheus, Grafana, Loki, Alloy configuration
docs/                          Deployment and codebase documentation
```

For route-level details and the interactive diagram, open the in-app handbook at `/docs`.
