DB: Hive, Impala, HBASE.
Data Processing: Spark core and SQL.
build tool: Maven , Testing framework: Cucumber
Required Skills :
· Over all 8 to 10 years of IT experience.
· Extensive experience in Big Data, Analytics, ETL technologies.
· Application Development background along with knowledge of Analytics libraries, statistical and big data computing libraries.
· Minimum 3+ years of experience in Spark/PySpark, Python/Scala/java programming.
· Hands-on experience in coding, designing and developing of complex data pipelines using big data technologies.
· Experience in developing applications on Big Data. Design and build highly scalable data pipelines.
· Expertise in Python, SQL Database, Spark, non-relational databases.
. Responsible to ingest data from files, streams and databases. Process the data using Spark, Python.
· Develop programs in PySpark and Python as part of data cleaning and processing.
· Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
· Develop efficient software code for multiple use cases leveraging Python and Big Data technologies for various use cases built on the platform.