Let's talk about Big Data and the data analyst profession

The subject Big Data is in vogue and it is no coincidence that society has been waiting for this type of solution for a long time and we just did not have the technology for it.

I'll try to explain the phenomenon by the end of this post:

We use Big Data for intricate diagnostics. This means that we use technology to learn more about natural phenomena, about organisms, about market trends, too complex subjects so that theorists can create accurate mathematical models that can be used by practitioners to detect a trend.

In the scientific environment, we call the inverse problem the attempt to understand a phenomenon from some peripheral measurements.

To solve an inverse problem you need to have data, and the more data you give, the more precise the modeling will be and the better the prediction will be. Yes, the name is even prediction!

But having data is not enough, it is necessary to have algorithms to find the model that best suits the phenomenon that one wishes to reproduce. And this is not a trivial problem, after all with the same data, several scenarios are possible, but they are not equally likely!

In this way, the algorithm search focuses on maximum probability methods, assuming that it will be the best solution for a set of measures or data. These algorithms use complex numerical methods, which are more accurately employed if the initial modeling takes into account the data obtained under different conditions.

To summarize: Big Data is a science that tries to create mathematical models about phenomena with many variables, models that will be used in prediction. Thus, a large volume of data, applied in probabilistic origin algorithms, using optimized numerical methods and an infrastructure capable of processing many operations in a short time, generates a model that will be used to predict (or even provoke) new behaviors.

Therefore, Big Data tools do not have as much power if we do not have an excellent body of data analysts, since information must be placed precisely at points of entry for the learning or prediction model to be worth.

For better understanding, it would be like trying to do a CT scan with a patient moving all the time, the result will not be good. Only in most real problems, we can not control the phenomenon as we do with a patient on a stretcher.

Applications of Big Data are numerous: from scientific research, market trends, public opinion, economics, meteorology to real-time supply chain analysis, sales forecasting, personalized marketing, etc.

Remembering that choosing a tool is only a small step to get results with this technology.

Among other companies, Lab245 offers data analysis and Big Data tool in cloud computing platform in Brazil.