Column-Level Lineage: Enhancing Data Accuracy and Governance

Experience profound metadata governance and enhanced data accuracy through column-level lineage with our detailed insights.

Jatin

October 3, 2023

Other blog articles

Reduce Time to Value for Data Observability

The revamped Config module is a new feature in our data observability solution that can help organizations to reduce time to value and get started with data observability quickly and easily.

Melanie Yong

September 29, 2023

Data Governance - Essential for the Modern Data Stack

Data governance is essential for modern businesses, but it can be a complex and time-consuming process. With this new update, you can store and manage all of your governance policies in one place, create custom classifications for data, and automate your data governance processes.

Melanie Yong

September 26, 2023

What is Data Observability? Data Engineers Guide

The technology landscape has shifted dramatically in the past few decades. As data becomes the new oil, its smooth flow, availability, and reliability are as essential as the oil flowing through the pipelines of the world. Just as the oil industry requires rigorous checks and measures to prevent leakages and ensure quality, the data industry requires its own set of checks, known as Data Observability. For data engineers, this emerging concept has become vital in ensuring that data systems remain robust, reliable, and trustworthy.

Jatin

September 25, 2023

Data Mesh: A Paradigm Shift in Managing Data at Scale

Data Mesh is revolutionizing the way data is managed in the digital age. In this blog post, we will explore what Data Mesh is and how it works. We'll also delve into the benefits of implementing Data Mesh as well as the challenges that come with it.

Jatin

September 25, 2023

Mastering Spark Jobs: A Comprehensive Guide for Data Engineers

Navigating the world of big data can be daunting, especially for newcomers. Our "Beginner's Guide for Spark Jobs" aims to simplify this journey by offering a comprehensive overview of how to effectively run and manage Apache Spark jobs.

Jatin

September 13, 2023

New: Incident Resolution with Enhanced Insights

An upgraded and focused experience within the Incidents Detail where you can see the metrics from each scan with the problematic points highlighted in the graph, assign a user to manage the incident, see the impacted areas from the downstream lineage, and even preview the affected rows.

Melanie Yong

September 13, 2023

Ensuring Data Quality thru robust Data Governance policies

This article emphasizes the paramount importance of implementing strong data governance strategies to guarantee the accuracy, reliability, and usability of data. The article discusses the risks posed by poor data quality, including misguided business decisions and resource wastage. By adopting robust data governance policies, organizations can streamline data management, enforce consistency, and ultimately drive better decision-making.

Jatin

August 14, 2023

Improving Data Team Performance through Data Observability

This article highlights enhancing data teams' efficiency by implementing observability. This approach provides insights into data systems, enabling quicker issue detection and resolution. The result is a more agile and responsive team, leading to improved performance, reliability, and overall success in data management.

Jatin

August 10, 2023

A Comprehensive Guide to Understanding Data Mesh Architecture

The blog provides an in-depth analysis of Data Mesh architecture, addressing its design, benefits, and implementation challenges. It emphasizes decentralized data ownership, enhancing operational efficiency and accessibility. The guide serves as an invaluable resource for enterprises aiming to leverage modern data strategies

Jatin

August 9, 2023

Data Observability: The Game-Changer in Maximizing the Accuracy of LLM Models

Data Observability is transforming the efficacy of Language Model Learning (LLM). It enhances data quality, improves model performance, and ultimately boosts accuracy. This game-changing approach equips data scientists to create robust, reliable models, revolutionizing the way we understand and implement machine learning technologies

Jatin

August 2, 2023

Discover with Ease: Enhanced Data Catalog with Semantic Search

See the power of our enhanced Data Catalog featuring Elasticsearch-powered semantic search. Effortlessly explore and find relevant tables with intuitive natural language queries and fine-tune results using in-built filters.

Melanie Yong

July 31, 2023

Exploring Data Lineage for Successful Data Governance Strategies

In today's data-driven world, organizations are generating and collecting vast amounts of data. However, without proper management and governance, this data can quickly become overwhelming and difficult to control. This is where data governance plays a crucial role. But what exactly is data governance?

Jatin

July 26, 2023

What is Graph Database? Use Cases and Advantages

In our information-rich era, traditional databases falter in managing complex relationships. Graph databases, gaining prominence for effectively handling connected data, provide a robust solution for multiple industries.

Jatin

July 25, 2023

Dagster versus Prefect

As data continues to grow exponentially, businesses are finding it increasingly difficult to manage and process it efficiently. One of the key challenges of working with large amounts of data is orchestrating it - ensuring that data is moved, processed, and analyzed in the right order and at the right time. This is where data orchestration tools come in. In this article, I'll explore the battle between two popular data orchestration tools - Dagster and Prefect - and help you choose the right one for your business.

Jatin

July 29, 2023

Gen AI and the future of Data engineering

Data engineering, the practice of collecting, transforming, and organizing data for analysis, is poised for a significant transformation with the advent of Generative Artificial Intelligence (Gen AI). Gen AI, a subfield of AI, focuses on creating AI systems that can generate new knowledge and insights. The future implications of Gen AI in data engineering are immense, promising to revolutionize the way we process, analyze, and utilize data.

Jatin

September 25, 2023

Building a Modern Data Stack: Discussion with Piyush Palkar, CDO of Carsome

In our next blog, we interview Piyush from Carsome, exploring his insights on constructing a Modern Data Stack. Expect discussions on cutting-edge data tools, strategies for data integration, and practical steps towards efficient data infrastructure. Don't miss this enlightening conversation that promises to upgrade your data proficiency.

Jatin

July 17, 2023

Develop a Robust Data Management Strategy for Effective Data Observability and Governance

In today's data-driven world, organizations are generating massive amounts of data at an unprecedented rate. This abundance of data presents both opportunities and challenges for businesses. To effectively harness the power of data, organizations need to develop a robust data management strategy. A data management strategy provides a framework for collecting, storing, organizing, and analyzing data to derive actionable insights and make informed decisions.

Jatin

July 13, 2023

Langchain : Concepts and getting started

Langchain is an open-source framework that facilitates the creation of large language model (LLM) based applications and chatbots. It provides a standard interface for interacting with LLMs, as well as a number of features that make it easier to build complex applications.

Jatin

July 13, 2023

What is Technical Metadata?

Discover the power of FinOps, the emerging discipline transforming cloud financial management. Learn how it optimizes costs, enhances collaboration between finance and operations, and enables efficient resource allocation. Explore key principles, benefits, and best practices in this insightful guide to FinOps.

Jatin Solanki

July 13, 2023

What is FinOps?

Discover the essence of FinOps in this concise article. Learn how it empowers organizations to optimize cloud costs, enhance financial accountability, and foster collaboration between finance and technical teams. Explore the benefits of implementing FinOps and unleash the potential for cost-effective cloud management.

Jatin Solanki

June 29, 2023

Training Language Models with Enterprise Data: Advantages and Considerations

This blog post provides a comprehensive guide to training language models with enterprise data. It explores the potential benefits and key considerations, such as data security, customization, scalability, maintenance, and cost. The post is a valuable resource for businesses considering integrating AI, offering practical tips and insights to make the process smoother and more effective.

Melanie

June 29, 2023

New connector: Azure Data Factory

Seamlessly incorporate Azure Data Factory pipelines into decube's comprehensive Data Catalog, monitor job statuses, investigate incidents, and visualize data lineage for optimized data transformation workflows.

Melanie Yong

June 26, 2023

Data Lake - Concepts

A data lake is a centralized repository that stores vast amounts of raw data in its native format. It allows organizations to store structured, semi-structured, and unstructured data, enabling flexible analysis and exploration. By eliminating data silos, data lakes promote data democratization and facilitate advanced analytics and machine learning initiatives.

Jatin Solanki

June 29, 2023

Importance of Embedding in AI / Generative AI

In the age of General Artificial Intelligence (GenAI), efficient data management is vital. Embeddings—a low-dimensional, vector representation of complex data types—offer a crucial pathway. They are enabling machines to understand and process diverse data more effectively. Coupled with the power of vector databases, embeddings enhance efficiency, scalability, flexibility, and interpretability, marking a paradigm shift in data engineering.

Melanie Yong

June 29, 2023

New: Group-Based Access Control and Approval Workflow

Introducing Group-Based Access Control and Approval Workflow! Take control of your user permissions and streamline access with Decube's latest upgrade. Learn more about our enhanced features and experience the power of tailored access control.

Melanie Yong

June 14, 2023

Migrating from Apache Airflow to Dagster

This blog post guides data engineers through migrating from Apache Airflow to Dagster, weighing the pros and cons of both platforms. It highlights the developer-friendly nature of Dagster, its support for data quality checks and strong data lineage, while noting the challenges of a less mature community and learning a new system.

Jatin Solanki

June 8, 2023

New Connector: Azure Synapse Analytics

Discover how the integration between Azure Synapse Analytics and Decube brings advanced data monitoring, in-depth insights and robust data governance. Get connected today and unlock a new level of data visibility, control, and insights.

Melanie Yong

June 8, 2023

What is ACID transactions?

ACID transactions are a fundamental concept in database systems, ensuring data integrity and consistency. The acronym stands for Atomicity, Consistency, Isolation, and Durability. These principles guarantee that all database transactions are processed reliably, even in the event of system failures, making them crucial for robust and secure data management

Jatin Solanki

June 29, 2023

What is MLOps?

MLOps, the fusion of Machine Learning and DevOps, streamlines the ML lifecycle, enhancing collaboration and communication between data scientists and operations teams. It ensures reliable, repeatable processes for model development, deployment, and maintenance. MLOps is the key to unlocking scalable, efficient, and robust AI capabilities in modern businesses.

Jatin Solanki

June 29, 2023

Streaming vs Batch Data processing

Batch and Streaming: two vital data processing approaches in today's digitized world. Our article delves into the intricate differences, comparing the mass data management of batching to real-time processing in streaming. Discover their unique pros, cons, and optimal usage scenarios, enabling strategic decision-making in your technology endeavours.

Jatin Solanki

June 29, 2023

What is Data Fabric?

Data fabric is an innovative approach to data management that connects diverse data sources, enabling seamless data access and processing. This network simplifies data integration, accelerates insights, and promotes business agility, driving the digital transformations of the future.

Jatin Solanki

June 29, 2023

Ingest data to Vector Database - Milvus example

This article guides you through ingesting data from AWS S3 into the Milvus vector database. It covers handling of diverse data types, from images to text and audio, by extracting feature vectors and storing them in Milvus for efficient similarity search. Detailed Python code examples are provided for data engineers.

Jatin Solanki

June 29, 2023

What is Vector Database? Concepts and examples

Vector databases, adept at handling complex, high-dimensional data, are revolutionizing data retrieval and analytics in the business world. Their efficiency in performing similarity searches makes them vital for applications like recommendation systems, semantic search, personalized marketing, and more, opening new avenues for data-driven decision making.

Jatin Solanki

September 25, 2023

Data Quality metrics for Data warehouse

Ensuring high-quality data is critical for any data warehouse. Our article explores key data quality metrics like accuracy, consistency, completeness, uniqueness, and timeliness. Learn how these metrics help maintain your warehouse's integrity, improve decision-making, and elevate your business's overall data strategy. Stay informed and ahead in your data-driven journey

Jatin Solanki

June 29, 2023

What is Data Freshness? Concepts

In the realm of data-driven decision making, data freshness is crucial. This article delves into the importance of timely, up-to-date data, the risks associated with stale data, and strategies for maintaining data freshness. Understanding and managing data freshness can help businesses make accurate decisions and maintain a competitive edge.

Jatin Solanki

June 29, 2023

New: Data Recon UI and Preview Sample Data

Data Recon gets a new look, and we've added the ability to preview sample data directly within decube. Check it out here.

Melanie Yong

May 19, 2023

Implementing Data contracts in the Data Warehouse

In the realm of warehousing, data contracts play a crucial role in establishing clear guidelines for data management and sharing. These contracts outline data ownership, usage rights, security measures, and compliance requirements, ensuring transparency and trust between stakeholders. Effective data contracts foster efficient operations and facilitate meaningful collaborations within the warehouse ecosystem.

Jatin Solanki

June 29, 2023

What is Data Profiling?

Data profiling is a crucial process in data analysis that involves examining and assessing datasets to gain insights into their quality, structure, and content. By understanding the characteristics and limitations of data, organizations can make informed decisions, ensure data integrity, and optimize data management strategies.

Jatin Solanki

June 29, 2023

Learn about Saudi Arabia's Personal Data Protection Law (PDPL)

In 2023, Saudi Arabia implemented a comprehensive Personal Data Protection Law aimed at safeguarding individuals' personal information. The law establishes guidelines for the collection, processing, and storage of data, grants individuals control over their data, and imposes penalties for non-compliance, ensuring enhanced privacy and data security.

Jatin Solanki

June 29, 2023

Introduction to Generative AI: Use Cases and Applications

Generative AI refers to the field of artificial intelligence that focuses on creating models capable of producing original and realistic content, such as images, music, and text. By leveraging deep learning techniques, generative AI opens doors to creative applications, but also raises ethical considerations regarding its potential misuse.

Jatin Solanki

June 29, 2023

What is Machine Learning Data Catalog?

Explore the power of Machine Learning Catalogs in this article, which delves into their role in streamlining data-driven decision-making. Discover how these catalogs revolutionize data discovery, governance, and collaboration by leveraging AI-driven insights, enabling organizations to unlock the full potential of their data assets.

Jatin Solanki

June 29, 2023

What is Apache Iceberg? Comparison with Parquet

Dive into the world of big data file formats as we compare Apache Iceberg and Apache Parquet. Iceberg's robust schema evolution, ACID transactions, and flexibility give it an edge, while Parquet's performance and ecosystem support make it a solid choice. Explore the nuances and decide which format suits you best!

Jatin Solanki

September 25, 2023

What is Metadata Management? Backbone of Data Catalog and Governance

Metadata management plays a pivotal role in data governance, providing a structured approach to understanding and controlling data assets. By capturing and organizing metadata, organizations can enhance data quality, ensure compliance, enable effective data lineage, and empower informed decision-making, fostering a robust foundation for data governance practices.

Jatin Solanki

June 29, 2023

Snowpark use case : For Snowflake users

Discover how Snowpark, Snowflake Data Warehouse's powerful developer experience, revolutionizes data engineering tasks by enabling complex data transformations using popular programming languages.

Jatin Solanki

June 29, 2023

Data Transformation use case : Apache Spark and Databricks

Explore complex data transformations using Apache Spark, and push the results to Databricks for analysis. Enhance data engineering workflows by leveraging these powerful big data tools.

Jatin Solanki

June 29, 2023

Data Engineers versus Software Engineers

Data engineers are often undervalued compared to software engineers, but their skills are critical in building data-driven businesses. This article advocates for recognizing their importance, highlights the impact of data engineering on products like ChatGPT, and suggests steps organizations can take to achieve parity between data engineers and software engineers.

Jatin Solanki

June 29, 2023

What is ClickHouse? Comprehensive Guide

ClickHouse is a columnar database management system specifically designed for online analytical processing (OLAP) workloads. It offers high-performance querying and analytics capabilities, making it ideal for handling large volumes of data with low latency. With its efficient compression and distributed architecture, ClickHouse is a powerful tool for real-time analytics and data-driven insights.

Jatin Solanki

June 29, 2023

A Day in Life as a Data Engineer | What they do?

A nerdy data engineer's perspective on a typical day and how their work influences the world of business

Jatin Solanki

June 29, 2023

What is SingleStore Database? Concepts and Importance

SingleStore DB is a powerful database management system designed to handle large-scale data processing and analytics. With its ability to support real-time applications, it offers high performance, scalability, and versatility. Its in-memory capabilities and distributed architecture make it a valuable tool for businesses seeking to optimize data processing and drive actionable insights.

Jatin Solanki

August 9, 2023

Data Pipeline Automation : Concepts and Techniques

Data pipeline automation is the key to achieving this. In this blog, we will explore the concepts and techniques involved in automating data pipelines, which can help you save time, reduce errors, and improve the overall data processing workflow.

Jatin Solanki

June 29, 2023

Data Transformation : clean, enrich and transform your data

Data transformation is a crucial step in preparing data for analysis. It involves three key processes: transforming raw data into a usable format, enriching the data by adding context or additional information, and cleaning the data by identifying and removing errors or inconsistencies. By applying these techniques, organizations can ensure their data is accurate, reliable, and ready for analysis

Jatin Solanki

June 29, 2023

Data Warehouse vs Data Lake : Understand the difference

Data warehouses are optimized for storing structured data in a highly organized manner, whereas data lakes can store both structured and unstructured data in a more flexible and scalable way. Data warehouses are best suited for analytics and reporting, while data lakes are better for exploratory analysis and data science.

Jatin Solanki

August 8, 2023

Data Observability - Importance and concepts for data engineers

Data observability is essential for data engineers, providing visibility into data freshness, quality, and lineage. It enables proactive issue resolution, improved data reliability, and enhanced collaboration. By implementing best practices and leveraging real-life case studies, data engineers can ensure data systems' optimal performance, drive accurate insights, and maintain compliance in today's data-driven world.

Jatin Solanki

June 29, 2023

Data Literacy : Why it its important for Data Engineers?

Data literacy is crucial for data engineers, enabling better decision-making, collaboration, innovation, and providing a competitive advantage. Developing data literacy involves continuous learning, hands-on experience, and staying updated with industry trends. Organizations play a vital role in promoting data literacy by creating a data-driven culture, offering training opportunities, and encouraging knowledge sharing.

Jatin Solanki

June 29, 2023

The importance of Data Lineage - Learn about upstream and downstream

Data lineage tracking is a critical aspect of data management that allows organizations to understand the origin and journey of their data. This blog discusses the importance of data lineage tracking, how it can benefit businesses, and some best practices for implementing it effectively.

Jatin Solanki

June 29, 2023

Newly launched: Airflow integration!

By popular demand, we’ve now released our latest integration with Apache Airflow, the open-source platform for programmatically authoring, scheduling, and monitoring workflows.

Melanie Yong

April 29, 2023

Data Mesh Architecture - Comprehensive Guide for Data Engineers

Data mesh architecture decentralizes data management, promoting domain-oriented ownership, treating data as a product, and encouraging self-serve infrastructure and federated governance. By understanding and implementing data mesh, data engineers can efficiently handle vast data volumes, driving innovation and value creation within their organizations.

Jatin Solanki

August 8, 2023

What is Data Science? Concepts and Guide

This article provides an overview of data science, its key components, impactful tools and technologies, and business and industry use cases. It also discusses the future scope of data science, including trends such as AI and machine learning, edge computing, cloud computing, and data privacy and security. The article concludes by emphasizing the importance of data science for driving innovation and growth.

Jatin Solanki

June 29, 2023

What is ETL? Difference between ETL and ELT

ETL and ELT are two approaches to data integration. ETL involves extracting data from multiple sources, transforming it into a compatible format, and then loading it into a target system. ELT, on the other hand, involves loading the data first and then transforming it. Both approaches have their advantages and drawbacks, and choosing the right one depends on the specific needs of the organization

Jatin Solanki

August 8, 2023

Data quality management : Tips & Best practices for reliable data

Data quality management is crucial to ensure accurate and reliable data. Best practices include establishing data standards, regular data audits, data validation, and ensuring data governance. Tips include using automation tools, involving stakeholders, and conducting regular data cleaning.

Jatin Solanki

June 29, 2023

Build Your Own Large Language model (LLM) Model with OpenAI using Microsoft Excel file

Discover how to build a custom LLM model using OpenAI and a large Excel dataset for tailored business responses. This guide covers dataset preparation, fine-tuning an OpenAI model, and generating human-like responses to business prompts. Boost productivity with a powerful tool for content generation, customer support, and data analysis.

Jatin Solanki

June 29, 2023

New: See the Lineage of your Transformations

Check out our update on Lineage, you can now see the lineage from dbt and Fivetran transformations!

Melanie Yong

April 27, 2023

Data Governance and Compliance - Beginner's Guide, Examples and Concepts

Data governance and compliance are critical for businesses to ensure the accuracy, availability, and security of their data. This beginner's guide covers the basics of data governance, compliance, and regulatory requirements, including key concepts, best practices, and tools for managing data effectively.

Jatin Solanki

June 29, 2023

Data Lineage : Examples, Concepts and Techniques

Data lineage refers to the process of tracking data from its origin to its destination, including all transformations and movements in between. It is crucial for data governance, regulatory compliance, and data quality assurance. Techniques for capturing data lineage include metadata management, data profiling, and data mapping.

Jatin Solanki

July 26, 2023

Defining Data Quality with SLA

Defining data quality with an SLA involves establishing metrics, monitoring, reporting, and remediation processes to ensure accuracy, completeness, consistency, timeliness, and validity of data. By defining expectations for data quality, organizations can improve decision-making and increase stakeholder confidence.

Jatin Solanki

June 29, 2023

New Integration Alert: Decube x SQL Server!

Exciting news for teams using SQL Server - we've just added a brand new integration! 🚀

Melanie Yong

April 13, 2023

Maximize the value of data with SingleStore and Decube's partnership

SingleStore and Decube have announced a partnership to provide an integrated solution for data observability, data catalog, and governance. The partnership aims to help organizations manage their data more effectively by providing a single platform for monitoring, tracking, and governing data across the enterprise.

Jatin Solanki

June 29, 2023

Data Dictionary : Guide, Examples, Best Practices and Templates

A data dictionary is a tool that provides a centralized repository for metadata about data. It defines the structure, format, and meaning of data elements, and serves as a reference for data analysts, developers, and other stakeholders. A well-maintained data dictionary can improve data quality, reduce errors, and promote better data governance.

Jatin Solanki

August 9, 2023

Learn SQL | Beginner's Guide to SQL

This beginner's guide to SQL covers the basics of creating, retrieving, updating, and deleting data from a database. Learn SQL guide also includes a section on joining tables together using inner, left, right, and full outer joins. With this guide, you can start exploring the vast world of SQL and using it to analyze and manipulate data.

Jatin Solanki

June 29, 2023

Two new integrations: Databricks and Fivetran

I’m thrilled to announce that we’ve added connections to Databricks AND Fivetran this week!

Melanie Yong

June 29, 2023

Rust - from PostgreSQL to Databricks using Diesel

In this article, we explore how Rust, Diesel ORM, and Databricks can be combined to create powerful data processing pipelines. We cover setting up a Rust project, configuring the database connection, running a query with Diesel, and processing data to Databricks using the Databricks REST API.

Jatin Solanki

July 17, 2023

Introducing the Business Glossary - A single standardised source of truth for important metrics

Start building your own dictionary for your organization, assign ownership to business metrics and link tables and columns to these metrics.

Melanie Yong

April 6, 2023

A Comprehensive Beginner's Guide to Databricks

The comprehensive guide to Databricks is an essential resource for anyone looking to leverage the power of Databricks for big data processing and analytics. The guide covers everything from getting started with Databricks to advanced topics like machine learning, deep learning, and natural language processing

Jatin Solanki

July 17, 2023

What is Metadata: Usecase, Definition, Types, and Importance

Metadata is information about other data that provides context and details about it. It is used to help organize, manage, and understand data in digital systems. Understanding metadata is important for ensuring proper organization and management of digital data.

Jatin Solanki

July 17, 2023

dbt - cheat sheet for data team

The post discusses how to create a dbt cheat sheet for data teams. The cheat sheet includes some of the most commonly used dbt commands and provides sample code snippets that data engineers can use as a reference when writing dbt code

Jatin Solanki

September 25, 2023

Apache Airflow vs Dagster - side by side Comparison

Apache Airflow and Dagster are open-source platforms for managing data workflows. Choose Apache Airflow for dynamic task generation and tool integration, and Dagster for strong data validation and ML framework integration.

Jatin Solanki

August 8, 2023

Why Data Catalog is crucial for Product teams?

Discover how data catalog can help product heads and managers improve data discoverability, understanding, governance, collaboration, and transparency. Learn how data catalog are like your personal assistant for data and how decube can help you take control of your data.

Jatin Solanki

July 17, 2023

What is Data Consistency? Definition, examples and best practice

Discover what data consistency means and why it matters for businesses. Learn how to ensure data accuracy and reliability.

Jatin Solanki

August 9, 2023

Upgrading your data catalog experience: Field Statistics and Schema

Check out new functionalities within Asset Details to add tags and classifications to fields. View statistics on your fields via the Field Statistics tab.

Melanie

April 6, 2023

A new connector to supercharge your observability: dbt!

We’re thrilled to announce that decube now can integrate with your dbt account. 🎉

Melanie

March 24, 2023

The synergies of Data Quality and Data Governance | Maximise value of data assets

Why it's important to deploy data quality and data governance to maximise the value of data assets

Jatin Solanki

July 17, 2023

Snowflake vs Databricks | Side by Side comparison

Snowflake is a cloud data platform, while Databricks is a data analytics platform. Both have their strengths and best suited for different use cases.

Jatin Solanki

September 25, 2023

Beginners guide for building data pipeline with Rust

This article introduces Rust, a systems programming language, and how it can be used to build data pipelines. It discusses the key features of Rust that make it well-suited for data engineering tasks, such as its performance, memory safety, and concurrency.

Jatin Solanki

July 17, 2023

What is Data Mesh? Architecture and best practice guide

Lets learn about data mesh architecture , examples, principles and about building a platform.

Jatin Solanki

September 25, 2023

Metric time selection is now optional for Google Big Query and Snowflake

Check out our important updates to the Data Quality engine.

Melanie

April 6, 2023

What is Data Catalog? Definitions, Example, Importance and Benefits

Lets drill down the basics of catalog and how it has evolved over the last few years.

Jatin Solanki

September 25, 2023

Introducing: Schema selection in Google Big Query, new looks in Asset Details and more!

decube now supports schema selection in Google Big Query so that you can choose specifically which schemas to view inside your Data Catalog. We've also changed up the look of Asset Details so you can add owners to your assets.

Melanie

April 1, 2023

Why Apache Airflow is not the best tool for data quality checks?

This article explains alternate options for Apache Airflow to manage or run data quality checks.

Jatin Solanki

July 17, 2023

Data Quality Guide to Accurate, Consistent and Reliable Data Management

Data quality is an important part of any organization, and poor data quality can lead to up to 60% of business decisions being based on inaccurate data. Implementing accurate data quality management systems can help the organization overcome these challenges.

Jatin Solanki

July 17, 2023

Top 5 Open source (free) ETL and Transformation Tools

ETL is one of the most critical pillar of a data-stack. We have curate top 5 open source (free) transformation tool for your benefits.

Jatin Solanki

September 25, 2023

What is Data Contracts, is it a hype?

Data contracts are formal agreements outlining the structure and type of data exchanged between systems, ensuring all parties understand the data's format. Used in various contexts such as APIs, SOA, data pipelines, they provide crucial interoperability, making data contracts essential in managing and controlling data flow effectively.

Jatin Solanki

September 25, 2023

What is Data Governance?

Making your company truly data-driven. This article will explain fundamentals and know-how in relation to Data Governance.

Jatin Solanki

September 25, 2023

What should I do after the Data Observability tool alerts me?

Steps you need to consider after you get an incident notification.

Jatin Solanki

July 13, 2023

What is Data Observability? A Comprehensive Guide

Data reliability is critical for data-driven organizations, but achieving it can be challenging. Implementing a data observability platform can help address these challenges by offering comprehensive solutions to improve data quality. By investing in a robust data foundation and adopting a data observability solution, businesses can make better-informed decisions and gain a competitive edge.

Jatin Solanki

September 25, 2023

Grow with our latest insights

Sneak peek from the data world.

Thank you! Your submission has been received!

Talk to a designer