WolframAlpha defines “Local Maximum” as a relative maximum somewhere in the neighborhood of the global maximum. Are you or your organization spending more and more resources and time on Advanced Analytics but you are not happy with the quality of the insights? This might be a sign that you have hit a local maximum.
How do you get out of this struggle zone? In this post let us look at couple of reasons.
Advanced Analytical Tools from your BI or Database Vendor : Your existing set of Database and BI tools are successful in transaction processing and Reporting. When this vendor provides advanced analytical tools which fit easily into your existing workflow, in general, these tools fail to deliver the expected insights. These tools are not the first choice for data scientist because they run inside the database, geared towards transactional processing and ELT. On top of it, they are expensive.
Most vendors provide Advanced Analytical tools in the database, but these are not specialized for data science. Their primary motivation is to prevent you from moving the data outside its data store. Embrace R and Python eco-systems. These tools are free, open source with a vibrant and active data science community. Support from an active community of experts is the key to success in Advance analytics.
Next, using Engineers to do analytical work: Your natural impulse is to use existing engineers or BI experts for analytics. Also, some data science tools deceptively look similar to engineering tools, and some even offer drag and drop workflows and configuration of hyperparameters. But the similarity end there. The world of engineers is deterministic, but data science embraces randomness. The engineer’s view of optimization is in terms of big-O-notation, but a scientist thinks in terms of optimizing the cost function. Both these projects have different lifecycles.
If you want deep insights, look beyond ETL, beyond heuristically describing the data and avoid focus on making the visualizations pretty. Try to find your answers using statistical analysis, discover the hidden networks and patterns in your data, augment the features from the structural data with the features from Text data, use NLP and Machine Learning.
Without understanding these difference some organization treat analytical projects as engineering projects and go to the extent of using not only engineers to do the analysis but also employ engineering project managers and the infamous agile methodologies. Ideally, it should be a combination of data scientists and engineers. Engineers help to operationalize and scale the model. Data science experts handle the modeling part of it.
To break away from the local maximum use tools specialized in data science like R and Python from Anaconda. Create a team comprising of both engineers and data scientists. Think about these project differently and start them differently than the traditional engineering projects.
We at Neocortex specialize in finding insights using advanced data analytics. We can help you get out of your local maximum. We can supplement your existing data science team or work with your engineering team for all your data science needs.