Data Processing

What does data processing offer to my company?

The databases of companies are a valuable source of information that correctly processed allows optimizing results and designing more effective business strategies.

Data processing is the area that stores, sorts, filters and process data.
The output resulting of the process is the data in a usable form.

Data Processing allows companies to obtain value information and a improved global overview.

Based on structured data we obtain valuable information that allows us to reduce costs, obtain more agile and reliable decision models, and detect new business areas or products.

OpenSistemas Processing area includes our products and projects related with technologies oriented to capture, transform, process and storage data, using ETL systems, big data platforms and NoSQL database technologies.
We are experts in acquisition and data storage solutions, NoSQL systems, ETL designing and both batch and lambda real-time processing over the cloud and Linux.

The core of our processing approach is the Apache Spark ecosystem, including Kappa and Lambda architectures. Different NoSQL databases, inmemory solutions and columnar databases together with ETL solutions for big data integration, like Pentaho Data Integration or IBM DataStage, complete our value proposal within this area. We include in this set different real time data processing solutions.

TECHNOLOGIES & TOOLS


  • Spark ecosystem: Spark, Scala, MLlib, GraphX, Spark Streaming, SparkSQL and Shark
  • Kappa architecture for Spark with Kafka, Spark, NOSql and Scala
  • Lambda architecture for Spark: Batch, Service and Speed Layers
  • Programing tecnologies for Big Data: Java, R, Python, C/C++
  • Other Big Data streaming solutions like Apache Storm or Kafka
  • ETL systems like Kettle ( Pentaho Data Integration ), DataStage and Talend ETL systems
  • OsBrain, the OpenSistemas Big Data Multi-agent platform
  • NoSQL Key-Value and Document Databases solutions like Cassandra or MongoDB
  • Columnar DataBases like MonetDB, SAP IQ, HBase
  • InMemory Databases like Redis
  • Other big data related techonologies: Hadoop HDFS, Hive/Pig, Flume, ElasticSearch
  • Big Data deployment over the cloud
  • Architecture design using Docker

APPLICATION AREAS


APPLICATION AREAS

  • Tax fraud environments
  • Electoral Systems- scrutiny and representation
  • Smartcities and smart environments
  • Financial markets Analysis and Trading
  • Massive data processing
  • Balanced Scorecards
  • Expert systems, predictive models and multiagent platforms

SUCCESS STORY


Big Data and Real Time data processing