Services

Data cleaning is one of the most important stages of preparing data, the aim of which is to cleanse input data and prepare it for analysis or storage.

Our transactional databases inevitably store incorrect, missing or redundant data. This is not a problem in transaction-oriented IT.

However, inadequately consolidated data can cause problems in business reports and analyses, or might generate even more serious issues in the case of forecasting or mathematical modelling.

In order to improve data quality, numerous techniques (for example, redundancy filtering, normalisation, and missing data replacement, among others) are used when consolidating a database.

Nowadays, data cleaning is a partially or fully automated process. As statistical reporting and analysis systems are already predominantly connected to transactional databases via APIs, it is also important to make data cleaning a real-time process.

Data cleaning – or rather consolidation – also involves the maintenance of the way data is interpreted. If, for example, a country was divided into five regions between 1990 and 2000, but this was changed to six regions between 2000 and 2010, the old code system would have to be adapted to correspond to the new code system, and vice versa.

Many of our corporate databases are designed and optimised to support operational processes. For instance, a ten-year loan agreement has a considerable amount of data associated with it, and it is important to be able to sort it according to its various aspects and on a single screen, if necessary. Similarly, the medical history of a patient or the project documentation of a construction site contains numerous branching points, as well as the related data and documents.

When we need to make a decision about a patient or a home, the answer is to be found in the operational systems. If we wish to gain an overview of patient groups or a new investment, we need to broaden our perspective. Although the data related to such questions also exists in our operational systems, it has to be summarised or other statistical indicators need to be introduced in order to provide more general answers.

If you frequently need to ask questions about your systems as a whole, and if your systems are multiple element structures whose elements are constantly changing, then you can only provide consistent answers over time with the help of a suitably designed data warehouse.

Building a data warehouse is a process in which data from different sources is collected in a single place, so as to make it readily available to decision-makers. When building a data warehouse, it is important to thoroughly consider the following questions during the design phase:

• What kind of data are we going to work with?
• How should we structure the data?
• What sort of queries do we need?
• How can we generate these queries efficiently and quickly?

Building and maintaining data warehouses is a dynamic process that is constantly evolving consistently with changing data and business needs. Be that as it may, the architecture and operation of data warehouses can significantly improve the soundness and efficiency of business decisions.

In addition to a data warehouse concept that was established decades ago, today there are also other formats that support analysis. A good example of this is a data lake, which is also a centralised collection of data/documents, but which, unlike a data warehouse, can also contain semi-structured or completely unstructured elements. Creating data lakes is a faster process requiring fewer resources, owing to the fact that these structures are primarily a working environment for professionals (such as data analysts and data scientists).

Az adattárházakban az adatokat tematikus, adott, a vállalatra szabott tárgykörök szerint rendezzük.

Az adattárházakban lévő adatok több forrásból származhatnak, mint például tranzakciós rendszerek, ügyfélkapcsolati rendszerek és egyéb külső adatforrások. Az adatok az adattárházba jellemzően adatfolyamokon keresztül juttatnak el, amely folyamat  lehetővé teszi, hogy az adatok valós időben kerüljenek át a vállalati rendszerbe.

Az adattárházakban az adatokat általában kimutatások, jelentések és BI dashboardok révén vizualizálják, ami megkönnyíti a döntéshozók számára az adatok értelmezését és a trendek azonosítását. Az adattárházak előnye, hogy az adatok konszolidáltak, strukturáltak és könnyen hozzáférhetők, ami segíti a hatékonyabb döntéshozatalt.

Az adattárház az adatokat szabványosított formára alakítva, egy helyre gyűjti és egységbe rendezve kezeli.

Az adattárházak építése komoly erőforrásokat igényel, beleértve az időt, a pénzt és az emberi erőforrásokat. A megfelelő tervezés és az adatmodell elkészítése azonban segíthet csökkenteni a költségeket és javítani a hatékonyságot.

Az adattárházak építése és karbantartása dinamikus folyamat, amely folyamatosan fejlődik az adatok és az üzleti igények változásainak megfelelően. Az adattárházak felépítése és működése azonban jelentősen javíthatja az üzleti döntések megalapozottságát és hatékonyságát.

Jó példa erre a folyamatra egy vállalat által épített adattárház, amelyben az összes ügyféladat, pénzügyi tranzakció, értékesítési adat és egyéb releváns információ tárolódik. Az adattárházat felhasználhatják az üzleti döntéshozatal során, például az értékesítési trendek elemzésére, a piaci igények felmérésére és a jövőbeli stratégiák kidolgozására.

The rocky road of digital transformation sooner or later brings us to the need for, and issue of, data(base) migration.

Database migration involves moving data from one database system to another. Migration can be triggered by a number of reasons, such as cutover to a new software version, the merging or splitting of IT systems, corporate consolidation or standardisation processes, among others. Nowadays, moving to the cloud is also often one of the reasons for migration.

Since, in addition to database structures, the construction of each database can also be different, it is likely that data consolidation steps will also need to be implemented. In other words, leveraging the benefits provided by the new structure (for example, data storage, compression, or indexing) also requires that the possibility to extract the original quantity of information from the new structure is ensured without any limitation.

Data migration comprises design, testing and implementation phases. It is important to note that migration may also have an impact on business processes due to databases being reorganised. Therefore the assessment of the design from the business intelligence aspect is also important for successful implementation.

During database migration, our experts will help you resolve any problems that may arise, as well as ensuring that the new database system is deployed to meet all your needs.

A document system describing data assets can be a crucial supporting tool for migration. It is essentially an inventory of information found in different systems, the respective data owners, update frequencies and hierarchies among elementary data. Having an inventory of data assets makes it substantially simpler to add interfaces to programs based on legacy systems.

An API (Application Programming Interface) is a tool that enables communication and data sharing between applications. APIs are fundamentally used for data communication among different systems. Using APIs increases interoperability between applications, and it also forms the basis for automatic data synchronisation.

In order to resolve API-related security issues, it is essential that data, and especially data channels, are properly encrypted and that API authentication procedures are properly managed.

APIs play a significant role in the digital transformation of businesses. APIs allow businesses to integrate different applications and services, which helps them make business processes more efficient and operation more flexible. Reporting interfaces (such as dashboards) usually do not extract input data directly from operational systems, but from a data warehouse or data lake. It could easily occur that the data warehouse is built and updated by a single API per incoming data source feed, and then a separate API is responsible for the data connection between the reporting interface and the data warehouse/lake.

APIs are of great significance in the integration of publicly available databases (for example, corporate data provided by Opten in Hungarian practice), in the development of online payment systems or, for instance, in automated complaint management or the development of chatbots.

With the advancement of Industry 4.0 methods, sensors capable of autonomous data transmission are proliferating exponentially. This type of data is also received via APIs. At present, given that procedures are not yet standardised, and, in some cases, due to the need to be cost-effective, there is still an extremely wide scope for API development related to automated metering/sensor technology

On its own, having the information is not enough: how we use the data we have and the knowledge we can gain from it is important.

Even the simplest bar or pie chart can help us understand the distribution or interactions of data. “A picture is worth a thousand words”, as the saying goes.

While in 2000 people could pay attention to something for an average of 12.5 seconds, a new study from Microsoft shows that today we only pay attention to content for an average of 8.25 seconds.

Providing a visualisation of the data is also important due to this drop in reception time and concentration, so that it can be consumed more quickly and effectively by our target audience, colleagues and partners.

There are two important areas of use for visualisation: revealing and interpreting data, on the one hand, and the visual representation of results, such as the results of mathematical modelling, on the other.

The effectiveness of the method is due to its ability to build dynamic reports from different data sources, such as Google Analytics, social media platforms, CRM systems, internal company systems, and proprietary databases, among others.

Data science-based visualisation can be used to make data easier to understand and more clearly arranged. For example, a company that collects data on its products and sales processes can create an interactive dashboard to display its data. This allows sales teams to examine data from different perspectives, such as by time, region or sales channel. With interactive visualisation, sales teams can easily identify sales opportunities and trends, which can help them make decisions. Data science-based visualisation enables the revealing of correlations, trends and patterns in data, which can help businesses operate more efficiently.

The role of forecasting in data science is of particular importance, as it allows business decisions to be substantiated by information regarding the future. Forecasts help companies plan and prepare for potential future challenges.

However, their preparation is a complex process that requires statistical and mathematical methods. Data must be prepared thoroughly, and appropriate data models must be developed. The credibility of the forecasts depends largely on the choice of appropriate data sources and the reliability of the data.

Multiple techniques can be used to generate forecasts, such as regression analysis, time series analysis, simulation and machine learning. Machine learning has recently resulted in a particular breakthrough in the field of forecasting, as it allows the analysis and prediction of highly complex data sets. What is more, with machine learning, forecasts become more accurate day-by-day without any further human intervention.

Specifying the error of estimation is an important feature of forecasts generated by mathematicians. The forecast, together with the specified reliability, thus becomes suitable for supporting actual decisions.

Forecasting can be done sector-neutrally, for selective purposes. It is possible to estimate cash flow available at a given moment, as well as anticipated reject rates, churn rate, the expected return on an investment or even the anticipated benefits of choosing a particular tax option subject to various development scenarios.

Business forecasts can help optimise production and increase efficiency. Using the data, businesses can forecast stock movements and procurement costs, which can help with financial planning. Taking their data as the basis, companies can easily identify the resources that have the greatest impact on the success of their business, and plan their investments accordingly. By analysing their data, companies are enabled to recognise potential problems in due time, and can prevent them from occurring, thus improving their business results.

With the increasing volume and variety of data, the application of data science has become a key enabler for improving business efficiency. One of the most important questions that can be answered by analysing data is whether the use of resources is optimal.

Data science solutions can be deployed in numerous areas, but the most important, after answering the above question, is the optimisation analysis work conducted in order to optimise the use of resources, besides automation and process control that implements the results of this work in processes.

In data science, optimisation can cover multiple areas, including data acquisition, data processing, modelling, and interpretation of results. Optimisation during data acquisition and preparation may improve the efficiency of the process, as well as reducing errors and redundancies. Optimisation can improve the speed and accuracy of data analysis by selecting the most appropriate algorithms and models during data processing.

The choice and fine-tuning of models can also have a significant impact on optimisation. Choosing the right models and fine-tuning the parameters allows algorithms to be made more efficiently, and increases the accuracy and reliability of results. Interpreting the results can help you make decisions that make your business processes more efficient.

Optimisation brings several benefits to businesses, including efficiency improvement, cost reduction, and gaining a competitive edge. A broad range of optimisation techniques are used in the field of data science, including linear programming, simulation, optimisation algorithms, and artificial intelligence.

There are countless examples of optimisation, such as call centre call optimisation, when operators are assigned to specific outgoing calls based on a historical indicator related to the operators. Mapping available mechanical and human resources to each other is likewise an optimisation procedure, again based on historical efficiency or quality indicators. Optimisation also occurs when processes are organised and controlled with an eye to minimising the movement/storage time of items used in production.

• Data science is the science of analysing and modelling reality using data. Data simulation is one of the most important tools in data science, making the modelling of real-world events possible before they happen.
• Data simulation involves fitting data into models that represent the real world. These models can be mathematical equations, simulation programs or artificial intelligence algorithms.
• Data simulation allows data to be used to answer questions such as “What would happen if…?” or “What impact could a change have on a given system?”. This approach has valid reason for being employed in a number of areas, such as transport, the economy, healthcare, and environment protection, among others.
• Data simulation also plays an important role in decision-making. Real-world modelling and comparing alternative decision options using statistical methods allows companies to predict potential ways of reacting to market changes, and how they can improve their operations. Simulation technology is not only limited to the business world. In the healthcare sector, for instance, it allows modelling how diseases spread, and facilitates the containment of epidemics.
• The software and tools required for data simulation are constantly evolving, and the increase in computing power allows more complex models to be run. Advances in artificial intelligence and machine learning offer further possibilities for data simulation and allow models to be made even more accurate.
• Data simulation is therefore an extremely useful tool for analysing data and modelling reality. The results thus obtained can help us better understand our world, predict the future and make more effective decisions.
• It is extremely important to understand, and, incidentally, this is not always understood, even by experts, that simulation techniques can be used to estimate relationships and functions between process components using quantitative methods, even if these relationships are not known to the modeller. The use of simulation technologies is therefore an excellent means of exploring and characterising processes.
• A data science az adatok felhasználásával a valóság elemzésének és modellezésének tudománya. Az adatszimuláció a data science egyik legfontosabb eszköze, amely lehetővé teszi a valóságban előforduló események modellezését, mielőtt azok bekövetkeznének.
• Az adatszimuláció során az adatokat olyan modellekbe illesztjük, amelyek a valós világot reprezentálják. Ezek a modellek lehetnek matematikai egyenletek, szimulációs programok vagy mesterséges intelligencia algoritmusok.
• Az adatszimuláció lehetővé teszi az adatok felhasználását olyan kérdések megválaszolására, mint például “Mi történne, ha…?” vagy “Milyen hatással lehet egy változás az adott rendszerre?”. Ez a tudományág hasznos lehet számos területen, mint például a közlekedés, a gazdaság, az egészségügy és a környezetvédelem.
• Az adatszimuláció fontos szerepet játszik a döntéshozatalban is. A valós világ modellezése és az alternatív döntési lehetőségek tesztelése lehetővé teszi a vállalatok számára, hogy előre lássák, hogyan reagálnak a piac változásaira, és hogyan javíthatják a működésüket.
• Az adatszimulációhoz szükség van nagy mennyiségű adatokra, amelyek segítségével a modellek pontosabbá tehetők. Az adatok elemzése és azok megfelelően formázott felhasználása lehetővé teszi az adatszimuláció eredményeinek megbízhatóságát.
• Az adatszimulációhoz szükséges szoftverek és eszközök folyamatosan fejlődnek, és a számítási teljesítmény növekedése lehetővé teszi a bonyolultabb modellek futtatását. A mesterséges intelligencia és a gépi tanulás fejlődése további lehetőségeket kínál az adatszimulációban, és lehetővé teszi a modellek még pontosabbá tételét.
• Az adatszimuláció használata azonban nem korlátozódik csak az üzleti világra. Az egészségügyben például lehetővé teszi a betegségek terjedésének modellezését, és segít a járványokkal szembeni védekezésben.
• Az adatszimuláció tehát egy rendkívül hasznos eszköz az adatok elemzésére és a valóság modellezésére. Az általa nyert eredmények segítségével jobban megérthetjük a világunkat, előre jelezhetjük a jövőt és hatékonyabban hozhatunk döntéseket.
• Az adatszimulációhoz szükséges adatok és modellek sokfélesége lehetővé teszi annak alkalmazását számos területen. Az üzleti világon kívül használják például a meteorológiai előrejelzéseknél is, hogy modellezze a jövőbeli időjárási viszonyokat, és segítsen előrejelzéseket adni.

When designing dashboards, several professional criteria need to be coordinated.

Planning starts from the demand side, which involves understanding the problem at the business level and translating it into the language of IT. At this point, business analysts play a key role.

Without interpreting and having a systemic understanding of the input data that is available, building a reporting environment that serves needs will not be possible. The data architect and ETL developer are responsible for these areas.

As concerns input data, we need to have an idea of data quality, availability, and update frequency, as well as the most important translation and link tables. Database administrators work together with ETL and BI developers on these tasks.

Not only must the dashboard be functionally complete, it also has to meet aesthetic or even company profile related requirements, and must serve a wide range of user needs. The design team and user experience team are responsible for designing the dashboard according to these criteria.

During dashboard implementation, an application is essentially developed in a suitable software environment. Essentially, we serve two dashboard environments:

Microsoft Power BI: primarily a data visualisation tool, which also allows the visualisation of modelling results through R (and Python) applications that can be integrated. MS Power BI has four great advantages:

the MS environment makes authentication a matter of course

also due to the MS environment, the linking of data with other MS data sources (for example, Excel) or applications (for example, Power Automate) is remarkably simple

usability on mobile devices requires no additional development or cost

price: if a company is an Office 365 user, Power BI is currently available without any additional payable license fees

Shiny/R: Data analysis and visualisation tool. Any data science problem can be solved in the R framework (some examples are presented on this link: (https://shiny.rstudio.com/gallery/). R-solutions can be easily integrated with external systems, so their use is ideal for solving even a single modelling problem.

There are a number of conditions that need to be met when operating the dashboards that have been created. Things that are or might be necessary include:

• a service provider with dependable know-how and a reliable server park
• setting the software environment to the appropriate security level
• bandwidth that is always appropriate and scalable, essentially for at least 99% availability
• continuous monitoring of log files and prompt intervention when necessary
• the necessary software updates, performed while continuous operation is maintained
• reliable and documented backup and archiving, on a case-by-case basis
• providing a backup system in the context of a Business Continuity Plan.

With more than twenty years of operational experience behind us, we have successfully tested and implemented a wide range of client to provider operating modes.

For example, we serve one of our largest customers through Amazon in Frankfurt, Germany, but we can also run our dashboard solutions in the cheapest virtual server environment that nonetheless meets the same expectations as above.