Fivetran expands SDK to simplify building custom data connectors
Fivetran has expanded its Connector SDK to enable custom connectors for any data source.
The update allows developers to build pipelines connecting even unique or internally developed systems, facilitating the centralisation of company data for analytics, artificial intelligence, and business decision-making.
With the Connector SDK, data teams now have the ability to build secure, reliable pipelines for a range of sources—from various applications and internal APIs to legacy systems. Developers write integration logic in Python, while Fivetran manages infrastructure elements such as deployment, orchestration, scaling, monitoring, and error handling. The process is designed to allow most connectors to be built and deployed within several hours, removing the need for DevOps support or dedicated infrastructure development.
Anjan Kundavaram, Chief Product Officer at Fivetran, discussed the approach companies often take when a prebuilt connector is unavailable. He stated: "When there isn't a prebuilt connector, most teams end up building and maintaining custom pipelines themselves. That DIY approach may seem flexible at first, but it often becomes a long-term burden with hidden costs in reliability, security, and maintenance. The Connector SDK changes that. Now, any engineer can build a custom connector for any source and run it with the same infrastructure, performance, and reliability as Fivetran's native connectors. It gives companies the flexibility they need without the tradeoffs."
The SDK offers the same infrastructure that supports Fivetran's managed connectors, handling automatic retries, monitoring, and alerting to ensure the accurate delivery of data to destinations such as BigQuery, Databricks, Snowflake, and other platforms.
Babacar Seck, Head of Data Integration at Saint-Gobain, shared his perspective on their experience with the Connector SDK. He said: "The SDK was a huge surprise in the best way. We expected to keep using Azure Data Factory for APIs because it was the only option. But once we saw what we could do with Fivetran's Connector SDK, everything changed. We can now build custom connectors in-house and respond to business needs much faster — all while seamlessly delivering data into Snowflake on Azure."
The company noted that the Connector SDK is being demonstrated to the public, with a focus on allowing data engineers to build custom connectors for moving data into cloud destinations tailored for analytics and artificial intelligence workloads.
Fivetran is known for working with organisations across various industries, enabling them to centralise data from software-as-a-service applications, databases, files, and additional sources into cloud destinations such as data lakes. The company's approach emphasises high-performance pipelines, security, and interoperability to help organisations enhance or modernise their data infrastructure.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Techday NZ
2 days ago
- Techday NZ
Snowflake boosts AI with real-time licensed content access
Snowflake has introduced Cortex Knowledge Extensions, allowing enterprises to supplement their AI agents with real-time, licensed content from third-party publishers, with Stack Overflow among the first partners to join the Snowflake Marketplace. The introduction of Cortex Knowledge Extensions enables enterprise customers to enrich their AI applications and agents with updated, reliable content from publishers such as Stack Overflow, USA TODAY, and Packt. This approach ensures proper attribution and licensing of content, distinguishing it from other systems that use scraped material without consent from original publishers. According to Snowflake, this new capability is designed to address challenges faced by both enterprises and publishers. Enterprises often struggle to gain access to timely external information for their AI systems, limiting accuracy and depth of insight. Meanwhile, publishers are seeking a secure and fair way to allow their content to be used by enterprise AI, with assurance of both compensation and control. "Building powerful AI apps and agents at scale hinges on enterprises having access to a wealth of internal and external data that adds rich context to AI outputs. Snowflake is raising the bar on enterprise-wide collaboration to make it even easier for customers to fuel their AI initiatives with AI-ready data and harness the power of agentic apps — regardless of whether the data and apps reside within their own four walls or come from trusted third-party sources. Our latest innovations enable teams to turn possibilities into reality with data and AI, all without worrying about security and governance risk," Prasanna Krishanan, Head of Apps & Collaboration and Horizon at Snowflake, commented on the launch. With Cortex Knowledge Extensions, publishers are able to list their content, such as news articles, textbooks, and research papers, on the Snowflake Marketplace. Enterprises can then purchase this content and integrate it into their AI-powered apps and agents, including Cortex Agents, Cortex Search, and the soon-to-be-available Snowflake Intelligence. This functionality enables AI systems to provide responses informed by timely and relevant information while allowing publishers to monetise their intellectual property under agreed licensing terms. The mechanism for delivering content through Cortex Knowledge Extensions relies on retrieval-augmented generation and is underpinned by Snowflake's Zero-ETL Sharing functionality. This setup empowers publishers to revoke access to content if necessary, while always displaying clear attribution and links to the original source, thereby enhancing reliability and provenance. Alongside Cortex Knowledge Extensions, Snowflake has introduced Semantic Model Sharing, which is currently in private preview. Semantic Model Sharing allows enterprises to integrate and interact with AI-ready structured data within their Snowflake Cortex AI applications — whether the data originates from internal sources or third-party providers. The use of semantic models helps ensure consistency in how data and business concepts are defined and applied across different systems, contributing to more trustworthy and accurate AI outputs. By mapping internal data to standardised semantic models, enterprises can accelerate insights, support more uniform decision-making, and access industry-standard metrics while maintaining governance and version control. Snowflake reports that these advances are intended to eliminate the manual effort required to create semantic models internally, while supporting high-quality, context-rich, and accurate AI responses. Users can directly interact with their data using Semantic Model Sharing in Cortex AI, including Cortex Analyst, Cortex Agents, and Snowflake Intelligence. In addition to content and model sharing, Snowflake is adding support for Agentic Native Apps in its marketplace. This feature provides customers with access to third-party agentic applications, which can securely combine provider and consumer data within the enterprise's governance framework. Data remains within the customer's environment while agents perform tasks such as portfolio management and optimisation, using proprietary algorithms and datasets. Currently, Snowflake Marketplace connects enterprises with over 750 providers, offering more than 3,000 live data, application, and AI products. The introduction of Agentic Native Apps is intended to give providers new ways to distribute and monetise their offerings while allowing enterprises to drive additional value from their data without compromising privacy or security.
NZ Herald
2 days ago
- NZ Herald
Wellington man gets shock $16,000 bill after using a Google AI-ready tool, Meridian culls 53 jobs as it offshores billing
Wellington tech contractor Drew Broadley says he racked up a surprise $15,760 bill during May using Google BigQuery, an online tool that's part of the Google Cloud Platform (GCP). The tech giant describes BigQuery as 'a fully managed, AI-ready data platform for managing and analysing data'. Broadley says it's
Techday NZ
3 days ago
- Techday NZ
Matillion launches Maia to automate & simplify data engineering
Matillion has introduced Maia, a suite of artificial intelligence-powered data agents designed to streamline data engineering workflows and enhance productivity for data professionals. Maia acts as a team of AI data engineers that supports the work of existing data professionals, spanning the entire spectrum of data engineering activities. The tool is positioned to ease the manual work typically associated with data pipeline creation, while also making these processes more accessible to users without deep technical expertise. According to Matillion, Maia agents enable the rapid development of complex, end-to-end data pipelines from natural language prompts. This capability is intended to accelerate project timelines and empower a broader range of users to engage directly in data engineering tasks. Matthew Scullion, Co-Founder and Chief Executive Officer at Matillion, commented: "Data engineering is centered on repetitive work that is important, but often gritty and sometimes boring. Imagine what could be achieved if the heavy-lifting of that gritty work was taken away so data engineers could focus instead on driving real business value and impact. That is exactly what Maia does - data engineering at the speed of thought." He added: "Maia takes AI capabilities in data engineering far beyond generating code, working end-to-end across the full workflow of data engineering, including ingestion, transformation, orchestration, data quality, DataOps, management and beyond." Building on Matillion's Data Productivity Cloud, Maia has been designed with cloud-based data and AI architectures in mind, with the aim of providing trust, speed, and scalability. The forthcoming availability of Maia through the Snowflake Marketplace is expected to expand access for users already utilising the Snowflake data ecosystem. Kieran Kennedy, Vice President, Data Cloud Products at Snowflake, said: "Maia allows joint Matillon and Snowflake users to do more with their data and AI, regardless of their technical capabilities. Its architecture means that the technology is trusted by design, providing users peace of mind alongside significant productivity gains. We look forward to seeing how users harness Maia to boost data productivity and free up data engineers to innovate and ideate." Beyond supporting human data teams, Maia also enables collaboration with other AI agents, which can independently request the creation of data pipelines for particular business objectives. This feature is intended to further reduce workflow bottlenecks and facilitate agile data management across various departments and business cases. Matillion has established a record of integrating AI functionalities into its platform, with Maia following a series of previous AI-focused releases. Earlier offerings began in 2023, with ongoing product efforts in 2024 encompassing integrations with Snowflake Cortex AI and Snowpark Container Services. The company is currently making Maia available by invitation, allowing selected users to request exclusive access and provide feedback during the early phase of roll-out. Matillion plans to gradually extend access to Maia for its broader user community as development progresses. Matillion's platform is used globally by enterprises across sectors including technology, finance, energy, and communications for a wide range of data-driven needs, from business analytics to machine learning and AI use cases. Maia is expected to expand these capabilities by automating repetitive work and broadening participation in data engineering.
