The Hadoop Data and Information specialist will be responsible to design, develop and support application solutions with focus primarily on Hadoop for a Financial Services Environment. This role will play a vital role in building new data pipelines from various structured and unstructured sources into Hadoop. Must be eager to wear multiple hats, and be capable of picking up new technologies at a fast pace.
ROLE KEY RESPONSIBILITIESStrong conceptual understanding of the context and business requirements. Should be able to understand the business needs, High Level design and produce Low level design documents, implement code in accordance with the best practices.Hadoop development and implementation.Loading from disparate data sets.Pre-processing using Hive and PigAbility to perform data quality checks in a methodical manner to understand how to accurately utilize client data.Expert level programming skills using Hadoop to meet the challenges of advanced data manipulation, complicated programming logic, and large data volumes is required.Ability to communicate results and methodology with the project team and clients. Should be able to work in offshore/onshore model.Ability to meet deadlines and thrive in a banking environment.Provides solutions for data driven applications involving large and complex data and providing reconciliation and test cases.Understand customer's Business processes and pain areas which need attentionSource system understanding and analysis.Solution Architecture for the entire flow from source to end reporting data marts.Design Conceptual and physical data model for a global datawarehouse.in the Hadoop world (ETL versus ELT)High Level & Low Level design for ETL Components in HadoopTest prototypes and oversee handover to operational teams.Propose best practices/standards.Hands on work on Sqooping, MapReduce, Hive transformations, combinersBuild monitoring and testing mechanisms around Sqooping and data transformationsContinuous improvements to the current Hadoop set up in terms of scalability, reliability and monitoring.Analyse and enhance the architecture of the current implementationBuild and manage customer relationshipsManage personal delivery on projects and enhancementsEnsure personal service level agreement standards are metImplement initiatives to improve application performanceEnsure quality of programming codeTranslate business requirements into system requirementsDesign and document robust, scalable solutions according to set standardsEnsure accuracy of code and adherence to requirementsEnsure all production changes are managed within the release cycleParticipate in the development of key standardsSeek new ways to optimise or innovate as it relates to the use of technologyEnsure personal adherence to agreed governance proceduresProactively identify and manage risksResponsible for the design and implementation of effective cross-functional business intelligence systems and processes.Analytical with an even mix of business acumen and technical capability.Responsible for working with analysts, managers, and executives to understand business needs and working with source owners to understand the data sources.An innate curiosity and some analytical capability with a passion for learning.Translate business and technical requirements into efficient sustainable solutionsBe able to do gap and impact analysis on the requirementAccept coaching and mentoring from senior developers/architectsDo QA on designs and development and completed projects as requiredComplete documentation of requirements and development according to defined standards.Deliver necessary documentations where required
EXPERIENCE AND QUALIFICATIONSTertiary qualifications with majors in at least one of the following: Computer Science, Information Systems or similarCertification in Hadoop Development.Strong experience in Hadoop – HIVE, Pig, Spark, Impala, Oozie, Sqoop, and Map ReduceWriting high-performance, reliable and maintainable code.Ability to write MapReduce jobs.Good knowledge of database structures, theories, principles, and practices.Ability to write Pig Latin scripts.Hands on experience in HiveQL.Familiarity with data loading tools like Flume, Sqoop and Kafka.Knowledge of workflow/schedulers like Oozie.Analytical and problem solving skills, applied to Big Data domainProven understanding with Hadoop, HBase, Hive, Pig, and HBase.Good aptitude in multi-threading and concurrency concepts.Must have Java experience.Financial Services experience.In-depth knowledge of Data Warehouse and Big Data best practices.Knowledge and technical appreciation of the interconnectivities and interfaces between various technical platforms, operating systems and processes.Good understanding of data ITILMust understand the need to align the IT and business strategies.