New: Data Recon UI and Preview Sample Data
Data Recon gets a new look, and we've added the ability to preview sample data directly within decube. Check it out here.
Since we launched decube last November, we’ve given our Data Quality and Catalog + Governance module each a facelift. So now, it’s time for Data Recon to get a new look!
If you haven’t tried out our Data Recon module before, it’s an easy way to compare and validate two versions of any datasets. This is important if your data is constantly being replicated and moved from one data warehouse to another, or going from staging and production. Discrepancies such as dropped or mutated data happen; and it’s often your team will end up spending precious time manually comparison between datasets with custom SQL queries, custom scripts or spreadsheets. But you don’t have to anymore.
With our Data Recon module, it’s easy to just select the tables you want to compare, configure the settings, set the schedule and let us do it for you.
A focused page for configuration
Creating a new data recon has been made into a focused experience where you can now set up your configuration without feeling cramped. We’ve also brought over the new asset icons from the Catalog and made it easier to search and find the tables from your data sources.
We’ve also updated the Data Recon Details page so that you have a focused view on the results of the recon. If you’d still like to see your Config that created the recon, you can check it on the top right button. That’s also where you can modify the schedule of the current recon if needed.
You can now also download the Details tab from your Recon in .csv format by using our "Export Recon" button in the top right. This will download all the rows that were found to be problematic during the recon job.
Compare between multiple data sources
You can now compare two tables between any supported data sources. That’s right, if you have to compare a table within PosgreSQL with one from BigQuery, you just need to toggle on “Use separate data sources”. You will just need to ensure that both tables share the same primary keys, and when adding the additional fields to be mapped, they are of the same logical type.
Filter rows by adding conditions
You can also add conditions directly to your recon configuration to help filter the rows to be checked. Use this to exclude rows that are not in the scope of the comparison, or set limits to the time period to be checked.
How this works is that we use your conditions as a WHERE clause. You just have to make sure that you write the condition in the SQL dialect of the source that you’ve selected. This is an optional step, and you can add them to either of your tables, both or none at all.
Make sure that your data arrives your destination in the way you expect them to be. Check out how to set up a new recon here.
Preview data directly from Catalog
Sometimes you just want to get a context of the table by seeing data directly from your table, and that’s fine. Now you can do that by navigating to the Preview tab for your table directly within decube’s Asset details itself to see 10 rows of sample data from your table.
Data that you have marked as PII or Sensitive in the Classifications of our Schema tab will be automatically showed as masked via our platform so that your sensitive data do not get exposed to your organization members.
Here’s a sneak peek for our readers: When we roll out our Group Access Control feature (very soon!), we’ll also provide the ability to limit access to who can see this tab, or run the Preview, so you’ll have further control over who can see the sample data.
Check out our Public Roadmap for what we're currently working on and what's going to be released real soon!