Hadoop Developer

Hadoop Developer
Company:

Nanyata Siyaya


Hadoop Developer

Details of the offer

ROLE KEY RESPONSIBILITIES® Strong conceptual understanding of the context and business requirements. Should be able to understand the business needs, High Level design and produce Low level design documents, implement code in accordance with the best practices.® Hadoop development and implementation.® Loading from disparate data sets.® Pre-processing using Hive® Ability to perform data quality checks in a methodical manner to understand how to accurately utilize client data.® Expert level programming skills using Hadoop to meet the challenges of advanced data manipulation, complicated programming logic, and large data volumes is required.® Ability to communicate results and methodology with the project team and clients. Should be able to work in offshore/onshore model.® Ability to meet deadlines and thrive in a banking environment.® Provides solutions for data driven applications involving large and complex data and providing reconciliation and test cases.® Understand customer''s Business processes and pain areas which need attention® Source system understanding and analysis.® Solution Architecture for the entire flow from source to end reporting data marts.® Design Conceptual and physical data model for a global datawarehouse.in the Hadoop world (ETL versus ELT)® High Level & Low Level design for ETL Components in Hadoop® Test prototypes and oversee handover to operational teams.® Propose best practices/standards.® Hands on work on Sqooping, MapReduce, Hive transformations, combiners® Build monitoring and testing mechanisms around Sqooping and data transformations® Continuous improvements to the current Hadoop set up in terms of scalability, reliability and monitoring.® Analyse and enhance the architecture of the current implementation® Manage personal delivery on projects and enhancements® Ensure personal service level agreement standards are met® Implement initiatives to improve application performance® Ensure quality of programming code® Translate business requirements into system requirements® Design and document robust, scalable solutions according to set standards® Ensure accuracy of code and adherence to requirements® Ensure all production changes are managed within the release cycle® Participate in the development of key standards® Seek new ways to optimise or innovate as it relates to the use of technology® Ensure personal adherence to agreed governance procedures® Proactively identify and manage risks® Responsible for the design and implementation of effective cross-functional business intelligence systems and processes.® Analytical with an even mix of business acumen and technical capability.® Responsible for working with analysts, managers, and executives to understand business needs and working with source owners to understand the data sources.® An innate curiosity and some analytical capability with a passion for learning.® Translate business and technical requirements into efficient sustainable solutions® Be able to do gap and impact analysis on the requirement® Accept coaching and mentoring from senior developers/architects® Do QA on designs and development and completed projects as required® Complete documentation of requirements and development according to defined standards.® Deliver necessary documentations where requiredTechnical Analysis and Design ® Analyse business requirements and translate them into technical and system requirements to best satisfy these requirements® Write and maintain technical documentation® Develop end to end Big Data solutionsUnit Testing ® Unit test own development® Review and Unit testing of Peer code® Must be able to analyse and diagnose problems independently but can seek help from others when necessary.Behavioural Skills ® Building and maintain relationships® Willing to learn® Have a collaborative mindset® Adopt a broad, long-term perspective® Have a strategic mindset when designing solutions that enable standardization and efficiency® Contribute to the success of the team; co-operate with other employees to improve personal and team performanceCompetencies & Attributes ® Team player & ability to work closely with others® Be able to function independently should that be required® Problem solver / solution provider® Passionate about BI and data integration® Committed, self-driven & highly motivated individual® Adaptable; be able to handle change effectively® Good communication skills® Should be available to work after hours® Able to work by themselves and in a team.Advisory Function® Provide technical assistance to Junior and intermediate developersSolution Providing® Design the solution® Design various solutions to ensure accessibility to data® Involved in project where specific data is required for business consumption® Design solutions that are flexible, scalable and reusable based on team coding standards and guidelines.® Conduct (where applicable) systems testing to ensure solutions are designed to end-user specifications.Data Management and MI Development® Perform data extractions from various sources according to specified business requirements.® Keep up to date with the data elements that are available within the company.® Investigate in accordance to business requirements, the availability of appropriate data to facilitate delivery.® Ensure appropriate application of error handling and data "clean-up" methodology to provide credible output.® Ensure data / source consistency and integrity® Perform data modelling exercises in the best way possible to improve efficiency.® Ensure that data quality controls are followed in the development lifecycle and throughout the data information lifecycle® Enforce coding standards, centralise data and reduce duplication.® Innovation value add where gaps are identifiedGovernance:® Ensure that governance is adhered to in the context of a Bid Data environment (specifically related to their working area)® Act as the review gate for code reviewsGeneral® Ensure knowledge is shared amongst peers through various mentoring initiatives® Ensure accurate and comprehensive documentation is established and maintainedProactive Team Support® Provide proactive, on-going support to cross functional teams and team members in order to support a mature work environmentEXPERIENCE AND QUALIFICATIONS® Tertiary qualifications with majors in at least one of the following: Computer Science, Information Systems or similar® Certification in Hadoop Development. (Advantageous)® In-depth knowledge of Data Warehouse and Big Data best practices.® Strong experience in Hadoop HDFS, HIVE, Spark, Sqoop, and Map Reduce® Writing high-performance, reliable and maintainable modular code.® Good knowledge of database structures, theories, principles, and practices.® Hands on experience in HiveQL.® Ability to use linux/unix OS® Practical experience using HDP or Cloudera® Knowledge of workflow/schedulers like Control M would be beneficial.® Analytical and problem-solving skills, applied to Big Data domain® Good aptitude in multi-threading and concurrency concepts. (Advantageous)® Must have python experience® Financial Services experience.® Knowledge and technical appreciation of the interconnectivities and interfaces between various technical platforms, operating systems and processes.® Good understanding of data ITIL® Must understand the need to align the IT and business strategies.

Requirements


Knowledges:

  • Python

Service engineer

Key responsibilities will include:Perform short term and site-based planning, coordination, and execution of planned maintenance.Coordinate and execute...


From Multitech Staffing Services - Mpumalanga

Published 18 days ago

Data engineer

ENVIRONMENT: A specialist IT Service Provider seeks the expertise of a proactive Data Engineer who is dogged about exceptional efficiency & eliminating waste...


From Datafin Recruitment - Western Cape

Published 24 days ago

Data engineer

ENVIRONMENT: A specialist IT Service Provider seeks the expertise of a proactive Data Engineer who is dogged about exceptional efficiency & eliminating waste...


From Datafin Recruitment - Western Cape

Published 25 days ago

Data engineer

ENVIRONMENT:A specialist IT Service Provider seeks the expertise of a proactive Data Engineer who is dogged about exceptional efficiency & eliminating waste and...


From Datafin Recruitment - Western Cape

Published 19 days ago