Overview
TDMQ for CKafka (CKafka) represents a scalable distributed messaging platform with high throughput, seamlessly compatible with Apache Kafka API versions 0.9 through 2.8. Operating on the publish/subscribe model, CKafka enables asynchronous interaction between producers and consumers, facilitating message decoupling. Its notable strengths include robust availability, efficient data compression, and support for offline/real-time data processing, rendering it well-suited for applications like compressed log aggregation, monitoring data consolidation, and stream data integration.
Benefits
Open-Source Component Compatibility
Boasting complete compatibility with Apache Kafka versions 0.9 through 2.8, our platform seamlessly integrates with upstream and downstream open-source elements like Kafka Streams and Kafka Connect. This integration effectively eradicates the expenses linked with cloud migration.
Upstream and Downstream Ecosystems
High Reliability
Exceed the efficiency of open-source alternatives and offer distributed deployment for enhanced cluster stability.
High Scalability
Facilitate automatic horizontal scaling of clusters and seamless instance upgrades, ensuring uninterrupted user experience.
Business Security
Implement network-level isolation for tenants across various accounts and enable CAM for managing streams and SASL for securing data streams, thus augmenting security measures.
Unified OPS Monitoring
Furnish a comprehensive suite of Operations (Ops) services, encompassing multifaceted monitoring and alert functionalities such as tenant segregation, access management, message retention querying, and consumer profiling.
Features
Message Decoupling
Peak Shifting
Sequential Read/Write
Async Communication
Scenarios
Log Analysis System
Streaming Data Processing Platform
Message storage
Data reporting and query
Database change subscription
Data integration
Data ETL and dumping
CKafka seamlessly integrates with EMR to establish a comprehensive log analysis system. Utilizing a client-side agent, logs are gathered and consolidated into CKafka. Here, the data undergoes iterative computation and consumption by backend big data solutions like Spark. Following this, the initial logs are refined, stored, or presented graphically as needed.
CKafka seamlessly integrates with Stream Compute Service (SCS) to analyze data in real-time or offline, enabling the detection of exceptions across various scenarios:
- Real-time analysis of data to identify exceptions, aiding in system issue diagnosis.
- Offline storage of historical consumption data for subsequent analysis and generation of trend reports.
By integrating SCF, CKafka enables tailored data processing to fulfill diverse message storage needs across various scenarios including log ingestion, microservices, and big data analytics.
The CKafka Connector offers extensive support for various data reporting scenarios, including analysis of operational behavior in mobile applications, logging bug reports on frontend pages, and reporting business data. Traditionally, such reported data needs to be transferred to downstream storage and analysis systems like Elasticsearch and HDFS for processing, which involves setting up servers, acquiring storage systems, and customizing code—an intricate and costly endeavor for long-term system operations.
Transitioning to a Software-as-a-Service (SaaS) model, the CKafka Connector streamlines this process into just two steps: configuration through the console and data reporting via the SDK. It is designed to be serverless and operates on a pay-as-you-go basis, eliminating the need for capacity estimation upfront and reducing development and operational costs.
By utilizing the CDC mechanism, CKafka Connector can efficiently subscribe to data modifications across various databases, including MySQL binlogs, MongoDB change streams, and row-level changes in PostgreSQL/SQL Server. In practical business scenarios, it’s often necessary to access MySQL binlogs to track changes (INSERT, UPDATE, DELETE, DDL, DML, etc.) and execute essential business logic operations such as querying, handling failures, and performing analyses.
However, constructing and maintaining these components can be resource-intensive. Additionally, a comprehensive monitoring system is essential to ensure the smooth operation of the subscription component.
In contrast, CKafka Connector offers SaaS components that streamline the process of data subscription, processing, and extraction through intuitive UI configurations.
CKafka Connector enables seamless integration of data from various sources (databases, middleware, log systems, application systems, etc.) across diverse environments (public cloud, on-premises data centers, cross-cloud, hybrid cloud) into CKafka for efficient processing and distribution. Typically, database data, business client data from applications, and log data require aggregation into a message queue for unified extraction, transformation, loading (ETL), analysis, and processing.
CKafka Connector provides robust capabilities for data aggregation, storage, processing, and dumping. In essence, it facilitates effortless data integration by connecting various data sources to downstream data targets.
In certain scenarios, data from a caching component like Kafka needs to be stored in downstream systems such as CKafka, ES, or COS following ETL processes. Traditionally, this involves using tools like Logstash, Flink, or custom code and requires monitoring to ensure stability. This approach necessitates understanding the syntax, specifications, and technical principles of these tools, leading to significant operational costs, especially for simple data processing tasks.
CKafka Connector offers a lightweight, UI-based solution for data ETL and dumping, simplifying the configuration process and enabling easier data processing and transfer to downstream storage systems.
Compute
Edge Computing
Container
Distributed Cloud
Microservice
Serverless
Essential Storage Service
Data Migration
Data Process and Analysis
Relational Database
NoSQL Database
Enterprise Distributed DBMS
Database SaaS Tool
Networking
CDN and Edge platform
Network Security
Data Security
Application Security
Endpoint Security
Business Security
Security Services
Security Management
Domains & Websites
Office Collaboration
Enterprise Applications
Data Analysis
Image Recognition
Face Recognition
Voice Technology
AI Platform Service
Natural Language Processing
Optical Character Recognition
Internet of Things
Message Queue
Middleware
Communication
Interactive Video Services
Stream Services
Media On-Demand
Media Process Services
Media SDK
Cloud Real-time Rendering
Game Services
Game Video Service
Education Services
Blockchain Service
Building Services
Cloud Resource Management
Management and Audit Tools
Developer Tools
Monitor and Operation
Education