Firefly Code Hierarchy
Table of contents
Use the following diagram to better understand the hierarchy amongst the core FireFly components, plugins and utility frameworks:
┌──────────┐ ┌───────────────┐
│ cmd ├──┤ firefly [Ff]│ - CLI entry point
└──────────┘ │ │ - Creates parent context
│ │ - Signal handling
└─────┬─────────┘
│
┌──────────┐ ┌─────┴─────────┐ - HTTP listener (Gorilla mux)
│ internal ├──┤ api [As]│ * TLS (SSL), CORS configuration etc.
└──────────┘ │ server │ * WS upgrade on same port
│ │ - REST route definitions
└─────┬─────────┘ * Simple routing logic only, all processing deferred to orchestrator
│
┌─────┴─────────┐ - REST route definition framework
│ openapi [Oa]│ * Standardizes Body, Path, Query, Filter semantics
│ spec | - OpenAPI 3.0 (Swagger) generation
└─────┬─────────┘ * Including Swagger. UI
│
┌─────┴─────────┐ - WebSocket server
│ [Ws]│ * Developer friendly JSON based protocol business app development
│ websockets │ * Reliable sequenced delivery
└─────┬─────────┘ * _Event interface [Ei] supports lower level integration with other compute frameworks/transports_
│
┌─────┴─────────┐ - Core data types
│ fftypes [Ft]│ * Used for API and Serialization
│ │ * APIs can mask fields on input via router definition
└─────┬─────────┘
│
┌─────┴─────────┐ - Core runtime server. Initializes and owns instances of:
│ [Or]│ * Components: Implement features
┌───────┬───┤ orchestrator │ * Plugins: Pluggable infrastructure services
│ │ │ │ - Exposes actions to router
│ │ └───────────────┘ * Processing starts here for all API calls
│ │
│ Components: Components do the heavy lifting within the engine
│ │
│ │ ┌───────────────┐ - Maintains a view of the entire network
│ ├───┤ network [Nm]│ * Integrates with network permissioning [NP] plugin
│ │ │ map │ * Integrates with broadcast plugin
│ │ └───────────────┘ * Handles hierarchy of member identity, node identity and signing identity
│ │
│ │ ┌───────────────┐ - Broadcast of data to all parties in the network
│ ├───┤ broadcast [Bm]│ * Implements dispatcher for batch component
│ │ │ manager | * Integrates with shared storage interface [Ps] plugin
│ │ └───────────────┘ * Integrates with blockchain interface [Bi] plugin
│ │
│ │ ┌───────────────┐ - Send private data to individual parties in the network
│ ├───┤ private [Pm]│ * Implements dispatcher for batch component
│ │ │ messaging | * Integrates with the data exchange [Dx] plugin
│ │ └──────┬────────┘ * Messages can be pinned and sequenced via the blockchain, or just sent
│ │ │
│ │ ┌──────┴────────┐ - Groups of parties, with isolated data and/or blockchains
│ │ │ group [Gm]│ * Integrates with data exchange [Dx] plugin
│ │ │ manager │ * Integrates with blockchain interface [Bi] plugin
│ │ └───────────────┘
│ │
│ │ ┌───────────────┐ - Private data management and validation
│ ├───┤ data [Dm]│ * Implements dispatcher for batch component
│ │ │ manager │ * Integrates with data exchange [Dx] plugin
│ │ └──────┬────────┘ * Integrates with blockchain interface [Bi] plugin
│ │ │
│ │ ┌──────┴────────┐ - JSON data shema management and validation (architecture extensible to XML and more)
│ │ │ json [Jv]│ * JSON Schema validation logic for outbound and inbound messages
│ │ │ validator │ * Schema propagatation
│ │ └──────┬────────┘ * Integrates with broadcast plugin
│ │ │
│ │ ┌──────┴────────┐ - Binary data addressable via ID or Hash
│ │ │ blobstore [Bs]│ * Integrates with data exchange [Dx] plugin
│ │ │ │ * Hashes data, and maintains mapping to payload references in blob storage
│ │ └───────────────┘ * Integrates with blockchain interface [Bi] plugin
│ │
│ │ ┌───────────────┐ - Private data management and validation
│ ├───┤ event [Em]│ * Implements dispatcher for batch component
│ │ │ manager │ * Integrates with data exchange [Dx] plugin
│ │ └──────┬────────┘ * Integrates with blockchain interface [Bi] plugin
│ │ │
│ │ ┌──────┴────────┐ - Handles incoming external data
│ │ │ [Ag]│ * Integrates with data exchange [Dx] plugin
│ │ │ aggregator │ * Integrates with shared storage interface [Ps] plugin
│ │ │ │ * Integrates with blockchain interface [Bi] plugin
│ │ │ │ - Ensures valid events are dispatched only once all data is available
│ │ └──────┬────────┘ * Context aware, to prevent block-the-world scenarios
│ │ │
│ │ ┌──────┴────────┐ - Subscription manager
│ │ │ [Sm]│ * Creation and management of subscriptions
│ │ │ subscription │ * Creation and management of subscriptions
│ │ │ manager │ * Message to Event matching logic
│ │ └──────┬────────┘
│ │ │
│ │ ┌──────┴────────┐ - Manages delivery of events to connected applications
│ │ │ event [Ed]│ * Integrates with data exchange [Dx] plugin
│ │ │ dispatcher │ * Integrates with blockchain interface [Bi] plugin
│ │ └───────────────┘
│ │
│ │ ┌───────────────┐ - Token operations
│ ├───┤ asset [Am]│ * NFT coupling with contexts
│ │ │ manager │ * Transfer coupling with data describing payment reason
│ │ │ │ - ...
│ │ └───────────────┘
│ │
│ │ ┌───────────────┐ - Aggregates messages and data, with rolled up hashes for pinning
│ ├───┤ batch [Ba]│ * Pluggable dispatchers
│ │ │ manager │ - Database decoupled from main-line API processing
│ │ │ │ * See architecture diagrams for more info on active/active sequencing
│ │ └──────┬────────┘ - Manages creation of batch processor instances
│ │ │
│ │ ┌──────┴────────┐ - Short lived agent spun up to assemble batches on demand
│ │ │ batch [Bp]│ * Coupled to an author+type of messages
│ │ │ processor │ - Builds batches of 100s messages for efficient pinning
│ │ │ │ * Aggregates messages and data, with rolled up hashes for pinning
│ │ └───────────────┘ - Shuts down automatically after a configurable inactivity period
│ ... more TBD
│
Plugins: Each plugin comprises a Go shim, plus a remote agent microservice runtime (if required)
│
│ ┌───────────────┐ - Blockchain Interface
├───────────┤ [Bi]│ * Transaction submission - including signing key management
│ │ blockchain │ * Event listening
│ │ interface │ * Standardized operations, and custom on-chain coupling
│ └─────┬─────────┘
│ │
│ ├─────────────────────┬───────────────────┬────────────────────┐
│ ┌─────┴─────────┐ ┌───────┴───────┐ ┌───────┴────────┐ ┌───────┴────────┐
│ │ ethereum │ │ corda │ │ fabric │ │ utdbql [1] │
│ └───────────────┘ └───────────────┘ └────────────────┘ └────────────────┘
│
│ ┌───────────────┐ - P2P Content Addresssed Filesystem
├───────────┤ shared [Si]│ * Payload upload / download
│ │ storage │ * Payload reference management
│ │ interface │
│ └─────┬─────────┘
│ │
│ ├───────── ... extensible to any shared storage sytem, accessible to all members
│ ┌─────┴─────────┐
│ │ ipfs │
│ └───────────────┘
│
│ ┌───────────────┐ - Private Data Exchange
├───────────┤ data [Dx]│ * Blob storage
│ │ exchange │ * Private secure messaging
│ └─────┬─────────┘ * Secure file transfer
│ │
│ ├─────────────────────┬────────── ... extensible to any private data exchange tech
│ ┌─────┴─────────┐ ┌───────┴───────┐
│ │ httpdirect │ │ kaleido │
│ └───────────────┘ └───────────────┘
│
│ ┌───────────────┐ - Identity resolution and mapping
├───────────┤ identity [Ii]│ * Resolves opaque identifiers used throughout FireFly
│ │ interface │ * Maps to and from blockchain signing identities
│ └─────┬─────────┘ * Map API/user identities from authentication, to network/organizational identities
│ │
│ ├───────────── ... extensible to DIDs etc.
│ ┌─────┴─────────┐
│ │ onchain │
│ └───────────────┘
│
│ ┌───────────────┐ - API Authentication and Authorization Interface
├───────────┤ api auth [Aa]│ * Authenticates security credentials (OpenID Connect id token JWTs etc.)
│ │ │ * Extracts API/user identity (for identity interface to map)
│ └─────┬─────────┘ * Enforcement point for fine grained API access control
│ │
│ ├─────────────────────┬────────── ... extensible other single sign-on technologies
│ ┌─────┴─────────┐ ┌───────┴───────┐
│ │ apikey │ │ jwt │
│ └───────────────┘ └───────────────┘
│
│ ┌───────────────┐ - Database Interactions
├───────────┤ database [Di]│ * Create, Read, Update, Delete (CRUD) actions
│ │ interace │ * Filtering and update definition interace
│ └─────┬─────────┘ * Migrations and Indexes
│ │
│ ├───────── ... extensible to NoSQL (CouchDB / MongoDB etc.)
│ ┌─────┴─────────┐
│ │ sqlcommon │
│ └─────┬─────────┘
│ ├─────────────────────┬───────────────────┐
│ ┌─────┴─────────┐ ┌───────┴───────┐ ┌───────┴────────┐
│ │ postgres │ │ ql │ │ sqlite │
│ └───────────────┘ └───────────────┘ └────────────────┘
│
│ ┌───────────────┐ - Connects the core event engine to external frameworks and applications
├───────────┤ event [Ei]│ * Supports long-lived (durable) and ephemeral event subscriptions
│ │ interface │ * Batching, filtering, all handled in core prior to transport
│ └─────┬─────────┘ * Interface supports connect-in (websocket) and connect-out (broker runtime style) plugins
│ │
│ ├───────── ... extensible to integrate off-chain compute framework (Hyperledger Avalon, TEE, ZKP, MPC etc.)
│ │ ... extensible to additional event delivery brokers/subsystems (Webhooks, Kafka, AMQP etc.)
│ ┌─────┴─────────┐
│ │ websockets │
│ └───────────────┘
│ ... more TBD
Additional utility framworks
┌───────────────┐ - REST API client
│ rest [Re]│ * Provides convenience and logging
│ client │ * Standardizes auth, config and retry logic
└───────────────┘ * Built on Resty
┌───────────────┐ - WebSocket client
│ wsclient [Wc]│ * Provides convenience and logging
│ │ * Standardizes auth, config and reconnect logic
└───────────────┘ * Built on Gorilla WebSockets
┌───────────────┐ - Translation framework
│ i18n [In]│ * Every translations must be added to `en_translations.json` - with an `FF10101` key
│ │ * Errors are wrapped, providing extra features from the `errors` package (stack etc.)
└───────────────┘ * Description translations also supported, such as OpenAPI description
┌───────────────┐ - Logging framework
│ log [Lo]│ * Logging framework (logrus) integrated with context based tagging
│ │ * Context is used throughout the code to pass API invocation context, and logging context
└───────────────┘ * Example: Every API call has an ID that can be traced, as well as a timeout
┌───────────────┐ - Configuration
│ config [Co]│ * File and Environment Variable based logging framework (viper)
│ │ * Primary config keys all defined centrally
└───────────────┘ * Plugins integrate by returning their config structure for unmarshaling (JSON tags)
[1] The "utdbql" blockchain plugin is a simple standalone ordering service, that uses the an in-process database
It does NOT provide a multi-party blockchain, and does NOT provide broadcast to all members in the network