FloodGate API Quick Start Guide

Overview

FloodGate is a RESTful API service that provides HTTP endpoints for DataFlood's document generation capabilities. It enables programmatic access to schema-based synthetic data generation, time-based sequencing, and batch operations.

Prerequisites

.NET 9.0 Runtime installed
DataFlood schemas (or sample data to create them)
Basic knowledge of REST APIs
API client (cURL, Postman, or programming language)

Starting FloodGate

Running the API Server

## Start FloodGate API server
cd FloodGate
FloodGate api

## API will be available at:
## http://localhost:5000

Running Tests (Optional)

## Run built-in test suite
cd FloodGate
FloodGate

## Tests verify all endpoints are working

Accessing Swagger Documentation

Open your browser to:

http://localhost:5000
or
http://localhost:5000/swagger

Swagger UI provides:

Interactive API documentation
Try-it-out functionality
Request/response examples
Schema definitions

Your First API Call

1. Health Check

Verify the API is running:

curl http://localhost:5000/api/documentgenerator/health

Response:

{
  "status": "Healthy",
  "timestamp": "2024-01-15T10:30:00Z",
  "version": "1.0.0"
}

2. Simple Document Generation

Generate documents with inline schema:

curl -X POST "http://localhost:5000/api/documentgenerator/generate-simple" \
  -H "Content-Type: application/json" \
  -d '{
    "schema": {
      "type": "object",
      "properties": {
        "id": {"type": "string", "pattern": "^ID-[0-9]{6}$"},
        "name": {"type": "string"},
        "age": {"type": "integer", "minimum": 18, "maximum": 65},
        "email": {"type": "string", "format": "email"}
      },
      "required": ["id", "name", "email"]
    },
    "count": 5
  }'

Response:

{
  "success": true,
  "generatedCount": 5,
  "documents": [
    {
      "id": "ID-123456",
      "name": "John Smith",
      "age": 34,
      "email": "john.smith@example.com"
    },
    ...
  ],
  "message": "Successfully generated 5 documents"
}

Common Use Cases

Generate from Uploaded Schema File

Upload a DataFlood schema and generate documents:

## Assuming you have a schema file: customer-schema.json
curl -X POST "http://localhost:5000/api/documentgenerator/generate-from-file" \
  -F "schemaFile=@customer-schema.json" \
  -F "count=10" \
  -F "format=json" \
  -F "seed=42"

Generate CSV Output

Generate data in CSV format:

curl -X POST "http://localhost:5000/api/documentgenerator/generate-simple" \
  -H "Content-Type: application/json" \
  -d '{
    "schema": {
      "type": "object",
      "properties": {
        "name": {"type": "string"},
        "value": {"type": "number"}
      }
    },
    "count": 100,
    "format": "csv"
  }'

Using Serving Mode

If you have models in the Serving folder:

## List available models
curl http://localhost:5000/api/serving/models

## Generate from a stored model
curl -X POST "http://localhost:5000/api/serving/generate/CustomerModel" \
  -H "Content-Type: application/json" \
  -d '{
    "count": 50,
    "seed": 12345
  }'

Setting Up Serving Folder

Create Serving Directory Structure

## In FloodGate directory
mkdir -p Serving/Models
mkdir -p Serving/Sequences

## Copy your schemas to Models folder
cp path/to/schemas/*.json Serving/Models/

## Copy sequences to Sequences folder
cp path/to/sequences/*.json Serving/Sequences/

Using Serving Endpoints

## List available models
curl http://localhost:5000/api/serving/models

## Response: ["CustomerModel", "ProductModel", "OrderModel"]

## Generate from stored model
curl -X POST "http://localhost:5000/api/serving/generate/CustomerModel" \
  -H "Content-Type: application/json" \
  -d '{"count": 10}'

Time-Based Sequences

Execute a Simple Sequence

curl -X POST "http://localhost:5000/api/sequencer/execute" \
  -H "Content-Type: application/json" \
  -d '{
    "configuration": {
      "name": "Test Sequence",
      "startTime": "2024-01-01T00:00:00Z",
      "endTime": "2024-01-01T00:05:00Z",
      "intervalMs": 10000,
      "steps": [
        {
          "stepId": "step1",
          "modelPath": "./Serving/Models/CustomerModel.json",
          "documentsPerInterval": 2,
          "generationProbability": 1.0
        }
      ]
    },
    "maxDocuments": 20
  }'

Client Examples

Python

import requests
import json

## API base URL
base_url = "http://localhost:5000"

## Generate documents
def generate_documents(schema, count=10):
    response = requests.post(
        f"{base_url}/api/documentgenerator/generate-simple",
        json={
            "schema": schema,
            "count": count
        }
    )
    return response.json()

## Example schema
schema = {
    "type": "object",
    "properties": {
        "id": {"type": "integer"},
        "name": {"type": "string"}
    }
}

## Generate and print
result = generate_documents(schema, 5)
print(f"Generated {result['generatedCount']} documents")
for doc in result['documents']:
    print(doc)

JavaScript/Node.js

const fetch = require('node-fetch');

const baseUrl = 'http://localhost:5000';

async function generateDocuments(schema, count = 10) {
    const response = await fetch(`${baseUrl}/api/documentgenerator/generate-simple`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({
            schema: schema,
            count: count
        })
    });
    
    return await response.json();
}

// Example usage
const schema = {
    type: 'object',
    properties: {
        id: { type: 'integer' },
        name: { type: 'string' }
    }
};

generateDocuments(schema, 5)
    .then(result => {
        console.log(`Generated ${result.generatedCount} documents`);
        result.documents.forEach(doc => console.log(doc));
    });

C#/.NET

using System.Net.Http;
using System.Text;
using System.Text.Json;

var client = new HttpClient();
var baseUrl = "http://localhost:5000";

var schema = new
{
    type = "object",
    properties = new
    {
        id = new { type = "integer" },
        name = new { type = "string" }
    }
};

var request = new
{
    schema = schema,
    count = 5
};

var json = JsonSerializer.Serialize(request);
var content = new StringContent(json, Encoding.UTF8, "application/json");

var response = await client.PostAsync(
    $"{baseUrl}/api/documentgenerator/generate-simple", 
    content
);

var result = await response.Content.ReadAsStringAsync();
Console.WriteLine(result);

Configuration

Using Configuration File

FloodGate can be configured via floodgate.config.yaml:

server:
  port: 5000
  maxRequestBodySize: 104857600  # 100MB

api:
  enableSwagger: true
  enableHealthChecks: true
  enableCors: true
  corsOrigins:
    - "http://localhost:3000"
    - "https://myapp.com"

serving:
  modelsPath: "./Serving/Models"
  sequencesPath: "./Serving/Sequences"

limits:
  maxDocumentsPerRequest: 10000
  maxConcurrentRequests: 100
  requestTimeout: 300  # seconds

logging:
  level: "Information"
  logToFile: true
  logFilePath: "./logs/floodgate.log"

Starting with Custom Configuration

FloodGate api --config my-config.yaml

Docker Deployment

Using Docker

## Build Docker image
docker build -t floodgate .

## Run container
docker run -p 5000:5000 floodgate

## With volume for serving folder
docker run -p 5000:5000 \
  -v $(pwd)/Serving:/app/Serving \
  floodgate

Docker Compose

version: '3.8'
services:
  floodgate:
    image: floodgate
    ports:
      - "5000:5000"
    volumes:
      - ./Serving:/app/Serving
      - ./logs:/app/logs
    environment:
      - ASPNETCORE_ENVIRONMENT=Production

Performance Tips

Batch Generation

For large datasets, use appropriate batch sizes:

## Good: Reasonable batch size
curl -X POST "http://localhost:5000/api/serving/generate/Model" \
  -d '{"count": 1000}'

## Better: Use streaming for very large datasets
curl -X POST "http://localhost:5000/api/serving/generate/Model/download" \
  -d '{"count": 100000, "format": "jsonl"}' \
  --output large-dataset.jsonl

Concurrent Requests

Use connection pooling for multiple requests:

import concurrent.futures
import requests

def generate_batch(batch_num):
    return requests.post(
        "http://localhost:5000/api/serving/generate/Model",
        json={"count": 100, "seed": batch_num}
    ).json()

## Generate in parallel
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    futures = [executor.submit(generate_batch, i) for i in range(10)]
    results = [f.result() for f in futures]

Monitoring

Health Endpoint

Monitor API health:

## Basic health check
curl http://localhost:5000/api/documentgenerator/health

## Detailed health (if configured)
curl http://localhost:5000/health/ready
curl http://localhost:5000/health/live

Metrics

If metrics are enabled:

curl http://localhost:5000/metrics

Security Considerations

API Key Authentication (if configured)

curl -X POST "http://localhost:5000/api/documentgenerator/generate-simple" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"schema": {...}, "count": 10}'

Rate Limiting

Default limits (configurable):

100 requests per minute per IP
10,000 documents per request
100MB max request size

Troubleshooting

Common Issues

Connection Refused

Verify FloodGate is running
Check port 5000 is not in use
Ensure firewall allows connection

Schema Validation Errors

Validate JSON syntax
Check required fields
Verify property types

Generation Failures

Reduce document count
Simplify schema
Check constraint compatibility

Performance Issues

Use appropriate batch sizes
Enable response compression
Consider using download endpoints

Next Steps

Explore API Reference for all endpoints
Learn about Serving Mode
Configure Sequences
Review Client Examples
Check Performance Guide

Getting Help

Check Swagger documentation at /swagger
Review error messages in responses
Enable debug logging for details
Consult Troubleshooting Guide