Skip to main content
Documentation Update

Garnet Framework has evolved with architectural improvements for better cost efficiency and scalability. This documentation is current and reflects these changes, though we continue to expand certain sections with additional examples and detailed guidance.

Thank you for your patience as we complete this update.

Building Context Producers

Context Producers are components that extract data from various sources and transform it into NGSI-LD compliant entities for ingestion into Garnet's knowledge graph. They serve as the bridge between your existing data sources and the unified contextual layer that Garnet provides.

Overview​

A Context Producer's primary responsibility is to take data from any accessible source and convert it into standardized NGSI-LD entities. These entities can then be delivered to Garnet through various methods, such as the SQS ingestion queue for batch processing or directly to the NGSI-LD Context Broker API for immediate updates. This transformation process enables data from disparate systems to become part of a unified knowledge graph where it can be correlated, analyzed, and consumed by various applications.

Context Producers operate independently from Context Consumers, allowing you to develop and deploy data ingestion capabilities without needing to modify existing applications or wait for consumer development. This decoupling accelerates development velocity and enables incremental system evolution.

Architecture Overview​

The Context Producer architecture consists of two main steps:

  • Data Ingestion: Extract data from the source system using the appropriate interface and technology
  • Data Transformation: Convert the raw data into NGSI-LD entities and deliver them to Garnet through the appropriate method (SQS ingestion queue for batch processing, direct NGSI-LD API calls for immediate updates, etc.)

Context Producer Architecture

You can build Context Producers using any technology stack, whether on-premises or in the cloud. Using AWS services provides access to a comprehensive set of tools including compute, database, IoT, analytics, and AI services that can enhance your Context Producer capabilities.

Mapping Data Sources to NGSI-LD Entities​

When building Context Producers, you need to make several key design decisions about how to map concepts from your data sources to NGSI-LD entity representations in the knowledge graph. This approach is flexible and spans across different domains and use cases. Here are some scenarios that illustrate the diversity of applications, though the possibilities are virtually limitless:

Scenario 1: Bike Sharing System
Your bike docking stations send status updates every few minutes about available bikes and docking spaces. Each update needs to consistently update the same BikeHireDockingStation entity, not create duplicates. The station ID from your bike sharing system becomes the key to maintaining this consistency - for example, if your data source provides station ID "BS001", you would use it as the unique identifier in the NGSI-LD URN: urn:ngsi-ld:BikeHireDockingStation:BS001.

Scenario 2: Customer Relationship Management
Your call center system generates call logs, customer interactions, and agent assignments. When a customer calls multiple times, each CallLog entity needs to link to the same Customer entity through consistent customer identification. Additionally, agents, tasks, and promotions all need to reference the correct customer entities to maintain relationship integrity.

Scenario 3: Agentic Systems
Your autonomous agents generate observations, make decisions, and use various tools. Each agent interaction needs to update the correct Agent entity while linking to the appropriate Tool entities they're using. The agent's unique identifier ensures all its activities are properly attributed and its knowledge base remains consistent.

Scenario 4: Multi-Source Integration
You have both real-time sensor data and periodic maintenance records for the same physical device. Both data sources should update the same Device entity using a consistent device identifier, regardless of which system provides the data first.

The core mapping process involves several key decisions:

Entity Identity: Establish consistent identifiers that map your data source IDs to NGSI-LD entity IDs. The same real-world concept should always map to the same entity identifier, regardless of which data source provides the information.

Entity Types: Choose entity types that create meaningful vocabularies for your applications. Consider using multi-typing (e.g., ["Sensor", "AirQualitySensor", "IndoorAirQualitySensor"]) to enable queries at different levels of specificity.

Relationships: Design connections between entities that reflect your domain's structure. Simple attributes can evolve into separate entities with relationships as your system grows in complexity.

For comprehensive guidance on entity identity, types, granularity, relationships, and advanced patterns, see Design Considerations.

Data Delivery Options​

Once you've transformed your data sources into NGSI-LD entities, you need to deliver them to Garnet. Context Producers can deliver transformed NGSI-LD entities to Garnet through multiple methods:

  • SQS Ingestion Queue: Uses SQS as a buffer for reliable delivery, particularly useful for high-volume data ingestion. The SQS ingestion performs batch upsert operations to the context broker with the options=update parameter to preserve existing attributes while updating only those provided in the payload
  • Direct NGSI-LD API: Direct API calls without buffering

The examples in this section primarily use the SQS ingestion queue for demonstration purposes. For detailed information about both delivery methods, including configuration and best practices, see Ingesting NGSI-LD Data and Using the NGSI-LD API.

Context Producer Examples​

Let's explore practical implementations for different data source types, demonstrating how to apply the entity linking concepts in real scenarios.

The following examples focus on the core concepts of data transformation and entity modeling. Implementation details like sendBatchToSQS() functions are simplified for clarity. For complete implementation guidance, including SQS batch message sending, IAM permissions, error handling, and other technical details, refer to Ingesting NGSI-LD Data and the AWS SQS documentation.

Fetching Data from External APIs​

When integrating with existing systems that expose APIs, you can use scheduled Lambda functions to periodically fetch and transform data. This pattern is ideal for systems that provide REST APIs with data you want to incorporate into your knowledge graph.

Architecture Overview​

The basic architecture consists of:

  • EventBridge Rule: Triggers Lambda function on a schedule
  • Lambda Function: Fetches data from external API and transforms it to NGSI-LD
  • SQS Queue: Receives transformed entities for Garnet ingestion

API Context Producer

Scenario: City Bike Sharing Integration​

Consider a city that wants to integrate real-time bike sharing data into their smart city platform. The bike sharing company provides a public API that returns current station status, but this data needs to be transformed into the city's unified knowledge graph to enable cross-system analytics and applications.

The challenge is that the bike sharing API returns data in their own format with their own identifiers, but the city's applications need this data to be consistently available in NGSI-LD format alongside other urban mobility data like bus stops, parking meters, and traffic sensors.

Implementation Example​

Here's how to build a Context Producer that periodically fetches bike sharing station data and transforms it for integration:

The Lambda function integrates with external APIs by:

  1. Making HTTP requests to fetch data from external systems (e.g., bike sharing API)
  2. Processing the API response and extracting relevant station data
  3. Transforming each station into NGSI-LD entities using consistent identifiers
  4. Using the stationId from the API as the unique identifier for consistent entity linking
  5. Sending all transformed entities to Garnet for ingestion

The stationId from the API provides the consistent identifier for linking all data from each station.

Creating API Endpoints for Data Reception​

When you need to receive data from external systems, Amazon API Gateway with Lambda integration provides a scalable solution.

Architecture Components​

  1. API Gateway: Exposes REST endpoint
  2. Lambda Function: Processes incoming data
  3. External Systems: Send data to your endpoint
  4. SQS Queue: Receives transformed entities

API Endpoint Context Producer

Scenario: Customer Profile Integration​

Consider a retail company that operates across multiple channels - mobile app, website, in-store kiosks, and loyalty program partners. Each touchpoint collects different aspects of customer information: the mobile app captures preferences and location data, the website tracks browsing behavior and purchase history, kiosks collect in-store interactions, and loyalty partners provide external purchase data.

The challenge is that each system sends customer updates in different formats and at different times. The API endpoint must normalize this data and ensure consistent customer identification across all touchpoints while providing reliable delivery to the unified customer knowledge graph for personalized experiences and analytics.

Security Considerations​

API endpoints that receive data from external systems must be properly secured to prevent unauthorized access and ensure data integrity. Consider implementing multiple security layers:

  • Authentication: Use API keys, IAM roles, or OAuth tokens to authenticate clients
  • Authorization: Implement fine-grained access control to limit what each client can do
  • Rate Limiting: Protect against abuse with throttling and usage plans
  • Input Validation: Validate all incoming data to prevent injection attacks
  • HTTPS Only: Ensure all communication is encrypted in transit

For comprehensive security guidance, see API Gateway Security Best Practices.

Implementation Example​

The Lambda function handles incoming API requests by:

  1. Parsing the JSON payload from the HTTP request body
  2. Extracting the customer identifier from the payload
  3. Transforming the received data into NGSI-LD entities
  4. Using the customerId as the unique identifier for consistent entity linking across touchpoints
  5. Sending the entity to Garnet for ingestion
  6. Returning appropriate HTTP responses to the client

This pattern enables multiple touchpoints to push customer data directly to your Context Producer through standard HTTP requests. Ensure proper authentication and authorization are configured at the API Gateway level.

Using AWS IoT Core for LoRaWAN​

AWS IoT Core for LoRaWAN is a fully managed LoRaWAN network server (LNS) that enables you to connect wireless devices that use the LoRaWAN protocol to AWS IoT Core. While this example uses AWS IoT Core for LoRaWAN, the same Context Producer pattern can be adapted for other LoRaWAN network servers by adjusting the data ingestion method and payload structure.

Architecture Components​

  1. LoRaWAN Devices: Send sensor data through LoRaWAN protocol
  2. AWS IoT Core for LoRaWAN: Receives and routes device messages
  3. IoT Rule: Routes messages to Lambda function
  4. Lambda Function: Decodes payload and creates NGSI-LD entities
  5. SQS Queue: Receives entities for Garnet ingestion

LoRaWAN Context Producer

Scenario: Smart Building Environmental Monitoring​

Consider a smart building management company that deploys LoRaWAN sensors throughout office buildings to monitor indoor air quality. These battery-powered sensors measure temperature, humidity, CO2 levels, and battery level, transmitting data every 15 minutes to optimize HVAC systems and ensure occupant comfort. Since the payload includes both environmental measurements and device status information, the Context Producer updates two separate entities: the sensor entity (with battery level and other device attributes) and an observation entity (with the environmental measurements), linked through relationships.

LoRaWAN payloads are limited in size, so sensors transmit only essential readings. However, the building management company can onboard entities with static information (like location and room placement) which then gets complemented dynamically by sensor readings from Context Producers. The knowledge graph aggregates both static and dynamic information around the same entities, creating a single source of truth that combines all available information. This enables comprehensive data retrieval through linked entity queries that can retrieve sensor details, room context, and measurements in a single request. For more information on linked entity retrieval, see Retrieving Linked Entities.

Implementation Steps​

  1. Device Registration: Onboard LoRaWAN devices to AWS IoT Core for LoRaWAN
  2. Device Onboarding to Garnet: Create sensor entities in Garnet with static information like location, serial number, installation date, room relationships, and other metadata that won't be transmitted through LoRaWAN payloads. Using the DevEUI (the LoRaWAN device identifier) in the entity ID provides a convenient way to link the static entity with dynamic updates from the Context Producer, since the DevEUI is included in the LoRaWAN payload
  3. Lambda Function: Create a function to process LoRaWAN messages and create NGSI-LD entities - this function decodes sensor payloads, transforms the data into NGSI-LD format, and sends entities to Garnet
  4. IoT Rule: Route device messages to the Lambda function
  5. Destination Assignment: Configure devices to use the IoT Rule

Lambda Function Example​

The Lambda function processes LoRaWAN messages by:

  1. Extracting the DevEUI (unique device identifier) from the message metadata
  2. Decoding the sensor-specific payload data
  3. Creating two NGSI-LD entities: one for the sensor (device-level attributes like battery level) and one for observations (measurement data)
  4. Establishing relationships through the measuredBy attribute linking observations to their sensor
  5. Sending both entities to Garnet's SQS ingestion queue

This demonstrates how a single Context Producer can update multiple entity types from one data source. The battery level belongs to the sensor entity, while temperature and humidity measurements belong to the observation entity, linked through the measuredBy relationship. The DevEUI provides the consistent identifier for both entities.

Using AWS IoT Core MQTT Broker​

For devices that communicate using MQTT protocol, AWS IoT Core provides a managed MQTT broker with built-in security and routing capabilities.

Architecture Components​

  1. MQTT Devices: Publish data to IoT Core topics
  2. AWS IoT Core: Manages device connections and message routing
  3. IoT Rule: Routes messages to Lambda function
  4. Lambda Function: Transforms messages to NGSI-LD
  5. SQS Queue: Receives entities for ingestion

MQTT Context Producer

Scenario: Industrial Equipment Monitoring​

Consider a manufacturing facility that needs to monitor critical machinery across multiple production lines. Each machine has vibration sensors attached to monitor engine health and detect early signs of mechanical issues. These sensors communicate via MQTT, sending vibration data that enables predictive maintenance algorithms to identify potential failures before they cause costly downtime.

The challenge is ensuring secure, reliable communication from hundreds of sensors while maintaining consistent device identification and proper relationships between sensors and the machines they monitor. Each sensor must be properly authenticated and its data must be linked to the correct machine entity in the maintenance system.

Implementation Steps​

  1. Device Registration: Create things in AWS IoT Core registry with meaningful names that align with your NGSI-LD entity naming convention. Garnet automatically syncs AWS IoT Thing lifecycle events (creation, deletion, group membership, connectivity status) as AwsIotThing entities through AWS IoT integration. Users can add static information like location and installation details to these entities (either before or after the sync creates them), including the placedIn relationship to attach sensors to machines. The Context Producer then updates these entities with sensor data and benefits from real-time connectivity status provided by the sync
  2. Certificate Management: Attach certificates and policies, ensuring each device uses its thing name as the client ID
  3. IoT Rule Creation: Route MQTT messages to Lambda function for processing
  4. Lambda Function: Process and transform messages using the thing name for consistent entity identification

IoT Rule Examples​

You have two main approaches for device identification:

If your device is already registered with AWS IoT Core, use the clientid() function to get the thing name directly. This works when the device connects using its thing name as the client ID, as described in the Associating an AWS IoT thing to an MQTT client connection documentation:

SELECT *, clientid() as thingName 
FROM 'sensors/+/data'

If you're setting up your own topic structure, you can include the thing name in the topic and extract it:

SELECT *, topic(2) as thingName 
FROM 'sensors/+/data'

Where the topic structure could be sensors/{thingName}/data.

The clientid() approach leverages AWS IoT Core's device registry and authentication, while the topic-based approach gives you flexibility in topic design. For detailed information about IoT SQL functions, see the AWS IoT SQL Reference.

Lambda Function Example​

The Lambda function processes MQTT messages by:

  1. Extracting the thing name using the clientid() function from the IoT Rule
  2. Accessing the MQTT payload data directly from the event
  3. Creating a VibrationSensor entity with only the sensor data (vibration level, frequency)
  4. Assuming the sensor entity was already created with static information including machine relationships
  5. Sending the sensor entity to Garnet for ingestion

This demonstrates how a Context Producer updates entities based only on the data the sensor provides, without establishing relationships that should be managed separately. The sensor entity is assumed to have been previously created with static information including its placedIn relationship to the machine. Applications needing holistic machine views can use subscriptions to aggregate sensor data and update machine entities with derived insights - for example, a predictive maintenance system could subscribe to vibration sensor updates and calculate overall machine health status based on multiple sensor readings. For more information on building such aggregated views, see Advanced Patterns: Aggregated Views.

Next Steps​

With Context Producers feeding data into your knowledge graph, you can now build Context Consumers that leverage this unified information. The next section covers how to create applications that consume and act upon the contextual intelligence your producers have created.

You can also explore practical examples in the tutorials section that demonstrate complete end-to-end implementations combining multiple Context Producers with various Context Consumers.