How Kissflow Breaks Down Silos: A Unified Data Strategy

Author: Adhi Ramanathan, AVP Product of Kissflow

Summary by M. Nivanya

Kissflow has emerged as a leading SaaS company, pioneering how organizations build and manage applications.  As a no-code, low-code platform, it empowers citizen developers to craft bespoke solutions without the complexities of traditional coding. 

However, as their customer base grew and the need for real-time data analytics emerged, Kissflow faced significant challenges that necessitated updating their architecture. 

This blog summarizes the presentation by Adhi Ramanathan, AVP Product of Kissflow at HTAP Summit 2024. It explores Kissflow’s journey from a traditional SQL-based system to a unified data strategy leveraging Hybrid Transactional and Analytical Processing (HTAP). In the presentation, he dives into the challenges faced during this transition, the solutions implemented, and the benefits realized. 

What Does Kissflow Do?

Kissflow is an innovative no-code, low-code platform designed to empower citizen developers—individuals who may not have formal coding skills but possess a keen understanding of their business needs. At its core, Kissflow simplifies the application-building process. Its user-friendly interface enables users to design workflows, forms, and reports through intuitive drag-and-drop functionalities. This approach not only speeds up development timelines but also fosters a culture of innovation by enabling non-technical users to take ownership of their processes.

Kissflow operates on a multi-tenant architecture, meaning multiple customers can utilize the same application instance while keeping their data secure and separate. This model is particularly useful for businesses looking to scale without incurring excessive overhead costs associated with managing individual instances.

The platform offers a variety of features and services tailored to meet the diverse needs of its users:

• Workflow Automation: Kissflow allows users to automate business processes, from simple task assignments to complex workflows, enhancing operational efficiency.
• Custom Forms: Users can create dynamic forms to capture specific data points relevant to their applications, facilitating better data collection and user interaction.
• Reporting and Analytics: The platform supports various reporting tools that provide users with insights into their data, helping them make informed decisions based on real-time information.
• Integration Capabilities: Kissflow can integrate seamlessly with other software and services, allowing businesses to connect their applications with existing tools and workflows.

Figure 1: Kissflow Form Builder 

Database Requirements for Kissflow

Several database requirements emerged as Kissflow evolved from a traditional SQL architecture to a more dynamic and scalable solution. These requirements were critical to ensure that the system could effectively support the diverse applications built by their users. The main database requirements included:

1. Transactional Data Storage: Kissflow needed a robust database capable of efficiently storing transactional data. This included all user-generated data from forms and workflows, such as purchase orders, user interactions, and application state changes. The database had to handle high volumes of transactions, especially during peak usage periods.
2. Application Metadata Storage: The platform needed a flexible solution to store user-defined application metadata for dynamic app creation.
3. Support for Complex Queries: Kissflow’s users often required the ability to run complex queries, including join operations for reporting and analytics. The database had to support advanced querying capabilities to generate insights from the data effectively. This was particularly important for users building custom reports or integrating with third-party BI tools.
4. Live Data Access: A critical requirement was providing customers with live data access. Many customers sought real-time access to their data to build complex reports using external analytics tools. This requirement needed a system that could support immediate data retrieval and updates, enabling users to see changes reflected in their dashboards without delay.
5. Cloud-Based Solution: As a SaaS platform, Kissflow required a cloud-based database solution. This ensured that the platform could scale effectively, manage resources efficiently, and provide high availability and reliability to its users. Cloud infrastructure also facilitated easier maintenance and reduced the burden of managing on-premises hardware.

Kissflow aimed to create a flexible, powerful, and user-friendly platform by addressing these requirements. It could adapt to its customers’ varying needs while maintaining high performance and reliability.

SQL and Initial Challenges

Kissflow began its journey in 2012, leveraging the capabilities of Google App Engine and Cloud SQL as the foundation of its platform. This early architecture was rooted in a traditional multi-instance SQL model, where each customer was provisioned with a separate database instance. This approach allowed for a clean separation of data and ensured that individual customer data remained secure and isolated.

Initially, this multi-instance architecture worked effectively, providing a straightforward way to manage applications for a limited number of customers. The benefits included:

• Data Isolation: Each customer’s data was kept separate, ensuring security and compliance with data privacy regulations. This was particularly important for organizations that needed to maintain strict control over their data.
• Simplicity: For a smaller customer base, managing individual SQL instances was relatively simple. It allowed Kissflow to provide tailored experiences and configurations specific to each customer’s needs.

The Challenges Faced

However, as Kissflow’s customer base began to expand rapidly, the limitations of this architecture became increasingly evident. The scaling demands of managing thousands of SQL instances introduced significant overhead and complexities, leading to several critical challenges:

1. Provisioning Overhead: Each time a new customer signed up, a new SQL instance had to be provisioned. This process was time-consuming and required considerable manual effort. Conversely, when a customer canceled their subscription, the deprovisioning process added further administrative burden.
2. Resource Management: Managing thousands of SQL instances required significant resources. Kissflow had to ensure that each instance was maintained, monitored, and secured, which became increasingly hard as their customer base grew.
3. Schema Management: When implementing schema changes or rolling out security patches, Kissflow needed to update thousands of SQL instances. This increased the risk of errors and led to delays in rolling out critical updates, potentially impacting the user experience.
4. Inflexibility in Data Management: The multi-instance architecture limited Kissflow’s ability to implement more dynamic features that could cater to varying customer needs. The rigid structure of SQL databases did not align well with the flexibility required by the diverse applications built on the platform.

Figure 2: Challenges in multi-instance SQL architecture 

As these challenges mounted, Kissflow realized its existing architecture could not sustain its desired growth and innovation. This prompted a critical evaluation of their database strategy and ultimately led to exploring more scalable and flexible solutions. In 2018, this resulted in an architectural shift, moving away from the constraints of a multi-instance SQL system to exploring NoSQL and data warehousing solutions.

Transition to NoSQL and Data Warehousing

By 2018, Kissflow realized that its existing multi-instance SQL architecture was no longer scalable enough to meet its evolving needs. The challenges of managing thousands of SQL instances and the increasing complexity of data handling prompted a complete re-architecture of its system. Kissflow turned to document-based NoSQL and data warehousing to address these issues, hoping to improve scalability, flexibility, and performance.

Figure 3: Solution with NoSQL and Data Warehouse

Why Did They Shift?

1. Flexibility of Schema: Kissflow’s platform allows users to create custom applications with unique structure and data requirements. This created a need for a flexible schema solution to handle dynamic application definitions. Unlike SQL databases with rigid schemas, NoSQL systems are built to handle varying and dynamic data structures. This made NoSQL the ideal choice for storing application metadata—allowing Kissflow to manage multiple, constantly changing schemas without the overhead of predefined structures.
2. Unified Storage for Transaction Data: Kissflow initially hoped that NoSQL could also be used as a unified database for transactional data and application metadata. Since NoSQL excels in managing unstructured data, it seemed like a good fit for their dynamic forms and transactional data, which varied from one customer to another.
3. Handling Reports and Analytics: Kissflow’s customers required sophisticated reporting and analytics capabilities, including complex queries such as join operations. Unfortunately, NoSQL systems, while excellent for flexible schemas, struggle with executing such complex queries efficiently. Recognizing this limitation, Kissflow introduced a data warehousing solution specifically for their analytical and reporting needs. This hybrid approach allowed Kissflow to utilize NoSQL for fast transactional data handling and metadata storage while leveraging data warehousing for reporting and live analytics.

The Challenges Faced

Despite the promise of this new architecture, the transition to NoSQL and data warehousing introduced new challenges that required careful navigation:

• Building a Custom ETL: To synchronize data between the NoSQL system (used for transactional data) and the OLAP (Online Analytical Processing) system in their data warehouse, Kissflow had to build a custom ETL (Extract, Transform, Load) pipeline. Off-the-shelf ETL solutions were not an option due to the dynamic schema in their applications—each customer’s application could have different fields and structures, requiring a high degree of customization.
• Latency in Data Synchronization: Data synchronization latency was one of the most significant challenges. The time taken to move data from NoSQL to the data warehouse could range from minutes to hours, which led to delays in reflecting updates in real-time dashboards. For customers expecting synchronous reports, this delay was a significant pain point. Users needed to see their form submissions or transactions reflected immediately in live reports, but the ETL process introduced unavoidable delays.
• High Maintenance of ETL Pipelines: Kissflow faced ongoing challenges in maintaining the ETL pipelines due to the complexity of dynamically changing schemas. Customers could modify their application structure anytime—adding new fields, changing data types, or altering workflows. This continuous schema evolution created ongoing maintenance issues, as the ETL job needed to be constantly adjusted to handle these changes smoothly. The team spent considerable time and resources to ensure the data synchronization remained stable and accurate.

While the combination of NoSQL and data warehousing improved scalability and provided flexibility for application definitions, the latency issues and high maintenance costs of the ETL process made it clear that this architecture had its limitations. The need for real-time analytics and seamless integration between transactional and analytical data pushed Kissflow to seek a more unified solution.

The Shift to HTAP: Enter TiDB

With the persistent challenges of maintaining separate systems for transactions and analytics, Kissflow searched for a more unified solution. Kissflow sought a database that could handle both transactional and analytical workloads in real-time, without the latency and overhead of custom ETL processes. This led them to explore HTAP solutions, which promised a unified system and a seamless user experience.

After evaluating several HTAP options, Kissflow chose TiDB, a powerful distributed SQL database designed for HTAP workloads. TiDB offered the best of both worlds: the transactional performance of traditional OLTP systems and the analytical capabilities of OLAP databases, all within a single platform. This shift allowed Kissflow to significantly simplify its architecture, consolidating most of its data management into TiDB.

However, NoSQL remained integral to Kissflow’s system, particularly for managing their dynamic metadata. Since every customer has unique application configurations and data structures, NoSQL’s flexibility made it the ideal choice for metadata storage. Instead of eliminating NoSQL, Kissflow built a unified interface integrating TiDB and NoSQL seamlessly. This unified interface eliminates the underlying database complexities, ensuring that services like workflow, reporting, and metadata management interact with the appropriate database without worrying about where the data is stored or how it’s retrieved.

This system’s heart is a data mapper that intelligently routes queries based on their purpose. If a service like reporting or workflow needs access to transactional or analytical data, the query is routed to TiDB. If the request is for application metadata, it’s directed to the NoSQL database. This unified session interface ensures the entire platform runs smoothly, providing a streamlined experience for developers and end users.

One key advantage of this new architecture is the elimination of the ETL latency that previously plagued Kissflow’s reporting modules. With TiDB handling transactions and analytics, reports are now updated in real-time, significantly enhancing the user experience. The company doesn’t have to deal with the complexities of maintaining custom ETL jobs. More importantly, its customers can now enjoy instantaneous insights and a faster platform.

Ultimately, the shift to TiDB marked a significant leap forward for Kissflow. Enabling it to deliver a more scalable, efficient, and user-friendly platform without compromising performance or flexibility.

Figure 4: New unified architecture of Kissflow 

A Unified Approach to Data with TiDB

Kissflow’s journey from traditional SQL, through NoSQL and data warehousing, to TiDB’s powerful distributed SQL architecture with HTAP components illustrates the importance of scalability, performance, and seamless data integration in SaaS environments. By embracing TiDB, Kissflow successfully eliminated data silos, reduced operational complexity, and delivered real-time insights—enhancing the overall user experience.

If your business is facing similar challenges managing transactional and analytical workloads or struggling with data synchronization delays, it might be time to explore a solution like TiDB. TiDB offers the flexibility, scalability, and power to handle complex, dynamic data requirements in real-time without the hassle of maintaining separate systems.

Ready to break down your own data silos? Reach out to the TiDB team today to schedule a demo or start a free trial!