Whether you work at a small business or Fortune 500 company, there is an agreed upon set of data that is used to run the business. CEO’s have an executive dashboard, Sales managers track leaderboards, and operations managers track Key Performance Indicators (KPI’s).
In a perfect world, each of these reports are timely and the data is reliable, consistent and accurate. In reality, there is likely a whole team of people who are straining to maintain these reports, and whose job it is to adapt to changing business needs. These people are the owners of your company’s Data Products.
It’s time to start thinking about data as a product. And unlike most products, anyone should be empowered to create data as a product!
What is a Data Product?
According to Tamr, “a Data Product is a consumption-ready set of high-quality, trustworthy, and accessible data that people across an organization can use to solve business challenges.” (Getting Started with Data Products: A CDO’s Guide)
At large organizations, there is likely a BI team who maintains data flows and outputs data into tools like Tableau, Power BI, or Qlik for consumption by the lines of business. At smaller organizations, there may just happen to be someone who was really good at Excel, who has become the de facto reporting person. Whether the Data Product owner has an official title or not, the role is being filled somewhere within the organization.
No matter your department, sales, marketing, or operations, in order to do your job, you require data. And the thing about data is that unless you are very, very lucky, the currently available Data Products are not going to contain exactly what you are looking for. Often, the desired level of detail is not available, but more likely, you need to combine one data set with one from another source.
Thus you wind up in an unenviable, but necessary position: a business person trying to create a Data Product. No, creating Data Products isn’t your day job, but this is the work required in order to do your job! It’s time to start thinking about data as a product.
In this blog, we are going to cover the common challenges involved with creating Data products and offer some solutions to help make your life easier.
5 Common Challenges with Creating Data Products
1. Data Products are hard to get prioritized, and ad hoc analysis will not be trusted
The moment you realized you had a data problem, you likely reached out to one of two people:
- BI Team - if your company has a dedicated team, it’s the right place to go. However, they likely have an intake and prioritization process that will slow this request down well beyond your deadline.
- That Excel Guru a few desks over - known for their prowess with spreadsheets, and offering quick results! But the outcomes may not be universally trusted and aren’t repeatable.
While both can get the job done, the timelines and mileage may vary. Obviously, we wouldn’t recommend skirting official channels, but the reality is that when you have a deadline to meet, you will figure out a way to hit it.
The Excel Guru is likely going to pull in data directly from a company database using Power Query, do some magic merging of files, and give you a spreadsheet for analysis. Just hope there were no mistakes made in the processes! It would be nicer, if you had a tool that YOU could use to explore data from the official source provided by the BI Team.
2. Data is too big for normal tools
Which brings us to our second point, if the data wasn’t so big you could just use tools that you are already familiar with, like Microsoft Excel or Google Sheets (both cap out at roughly 1 million rows!). But the data is big, because everything is measured. And if you only cared about summarized data, you wouldn’t be trying to create your own Data Product. You need the details!
An easy to use tool to explore the data would be nice. As the master of your craft, your job isn’t to also run big data software. We once had a customer tell us, “I can’t read music, but I know a bad note when I hear it.” You probably feel the same way about your business data. You just need to see it!
3. Data needs to be enriched with data from other sources
Data is like a fine wine, it’s so much better when paired with the right course. Besides the details, the other reason you are creating a new Data Product is that there is this one column from over there that MUST be included in order to provide the context needed for your business.
The Excel Guru can perform that V-Lookup, but if the data gets too big, it won’t work at scale. You need a tool that can JOIN data, as it’s referred to in database circles, from other sources.
Tools also exist that can use API’s to go out and gather external data to enrich your source data. For example, validating an email address, or geo locating an IP address to provide city and state information.
4. Data Developers are not Business Users
Let’s say you have been waiting a month for the BI team to finally provide you with your report. You open it up and it’s not what you were expecting. The data is grouped completely differently than you were expecting and the report is useless! How could that happen, when you clearly laid out the requirements. You even gave them a pivot table of sample data and created the exact charts required.
It would be easier if there was a tool that allowed you and the BI Team to iterate on analysis of the actual data as it’s being built, rather than gathering requirements and delivering something unexpected. In a perfect world, you would get the data exactly how you want it, and then say “automate this!”
5. Data Goes Bad
Which brings us to the final challenge of Data Product creation. Like you favorite food, data spoils. While you may have outsmarted the BI team and had the Excel Guru make you a questionable file, is that person going to be available to do it month after month? You called in a favor, but they did not sign up for a full time job. That’s what the BI Team is for!
Data goes bad and analysis needs to be repeated on a monthly basis. The enrichments need to be run again and data must be checked to ensure it us up to date. That’s where companies like Tamr come in, doing the data work for you so that you can run your business.
Tools to make Data Product Development Easier
That said, you are going to go forward and create your Data Product, because if you don’t do it, who else is? Your business is only going to get better if you make it happen. If you haven’t thought about your reports as Data Products, maybe it’s time you start thinking of data as a product.
And if you are looking for a way to manipulate big data, but are not a programmer, let me introduce you to Gigasheet, the big data spreadsheet that is democratizing big data. You are never going to have to bother the Excel Guru again! If you can use Excel or Google Sheets, you can use Gigasheet!
Let’s walk through the 5 challenges again and show how Gigasheet can help, for free!
1. Build Your Own Data Products
Connect Gigasheet directly to the trusted company data source, or cloud drive, and load up to a billion rows of data into a sheet. You are no longer constrained by the limits traditional spreadsheets and are able to perform that magic that the Excel Guru performed.
Once open, sort, data, use multiple filters, groups, and pivot tables just like you would in a spreadsheet. The data will match the gold standard of the BI team, because it is their data set. There is nothing to explain as to how you got the answers you came up with, so it will be trusted!
2. Too Big for Excel or Sheets? No Problem!
Beyond the row limits of Microsoft Excel, Google Sheets, or airtable, there is nothing worse than working with big files. How do you share the analysis with your team? You can’t email the file, and if it’s a large file people will still have to download it from a link and then open it.
With Gigasheet, makes data delivery easy because you can securely invite users to collaborate in the cloud. They can view your work and even make copies for their own analysis, without ever downloading or uploading a data set.
Additionally, if you want to publish a data set to the world, simply mark a file as Public, and then embed spreadsheet on your website or share the link on social media. People will be able to explore the data without even logging into Gigasheet.
3. Create Curated Data with Rich Context
Gigasheet can perform the trusty V-Lookup function on files with a Billion rows. Let’s say you acquire a clean data file from Tamr and want to compare it to your company’s source data. This is easy to do using V-lookup, just like in your favorite spreadsheet software. Watch the video below:
You can also use Gigasheet to run data enrichment API’s on your sheet without writing code. Gigasheet has built in enrichments for email and IP addresses. However, you can also use Gigasheet to run any available API by using the Custom Enrichment functionality.
4. Requirements Collaboration On Real Data
By using Gigasheet to allow Business Users to get the data exactly how they want it, BI Teams can accelerate the development process. This eliminates any confusion between requirements and report development because the users can work directly on the actual data. Once the data is in the right format, the BI team can write scripts to automate the data processing and deliver data that answers the right questions.
5. Quickly Refresh Stale Data
After processing and enriching source data, a data can be delivered with Gigasheet to end users using sharing. That way they will always have the most accurate data to run their business.
Final Thoughts on Data as a Product
Data Product Development can be challenging, but the first step to developing a good Data Product is to realize that you are developing a product! If done correctly, it's a high-quality, trustworthy, and accessible report that people across an organization can use to solve business challenges. While this may not be in your job description, it’s something that you must do and tools like Gigasheet can help Data Product development easier.
Sign up and try it now for free!
The ease of a spreadsheet with the power of a data warehouse.
