Build the foundations behind data / TWA · Enugu

Data Engineering:
make data dependable before it reaches a dashboard.

A report can only be as useful as the information behind it. Explore how data is collected, transformed, checked and delivered so other people can work with it reliably.

A pipeline should explain the second run

  1. Ingest
  2. Transform
  3. Check
  4. Serve

The two scenarios reveal reliability decisions; no database or live file is connected.

A pipeline is a responsibility, not just a transfer

Imagine receiving a daily order file from a fictional shop. Moving it into a database is a start. A dependable process also asks whether the file arrived, whether the columns changed, whether an order appeared twice and whether a failed run can be retried without duplicating records.

Data engineers build and maintain these paths between sources and usable datasets. Their work connects producers of information with analysts, applications and other consumers. A successful file copy does not establish that the data is complete or ready for a decision.

Understand records and meaning before adding orchestration

Foundations include SQL, tables, keys, data types and how records relate. Python can help read, transform and inspect data. A data model explains what a row represents: an order, an order line or an event. Confusing those grains can multiply totals when tables are joined.

TWA’s existing direction includes relational databases, modelling, Python, ETL/ELT concepts, quality checks, documentation and cloud storage/warehouse concepts. Exact database, orchestration or cloud products are not confirmed. Apache Airflow’s documentation is industry reading about task dependencies and workflow execution, not a promise that Airflow is included.

The TWA practice studio / Guided first steps

Make a retry safe for the sales report.

A shop sends orders A and B to a reporting table. A connection interruption means the same batch may be delivered again.

Your role: Run the same orders twice, inspect the repeated copies, then use stable order labels to prevent duplication in a clean reporting table.

What you’ll learn: Keep a repeated delivery from becoming duplicate sales in a report.

Read the full guided story without the interactive controls

You can follow the same lesson here without JavaScript.

  1. Load the first batch

    Copy orders A=120 and B=80 into the empty reporting table.

    What happens and why: Two rows produce a sum of 200 units. Copying received rows to the end of a table is called appending.

  2. Retry the same delivery

    Run the same batch again and compare the reporting rows and sum.

    What happens and why: Appending the retry creates A, B, A, B and a sum of 400. These are repeated records of the same orders, not new sales.

  3. Design a destination with a stable key

    Start a clean destination with order ID as its key. This explicitly rebuilds the toy table; it does not magically repair an existing production table.

    What happens and why: An order ID identifies the business record across retries. A run number identifies the attempt, so using it as the key would not recognise a repeated order.

  4. Rerun against the keyed destination

    Load and retry using “insert new IDs; update existing IDs.” Watch whether the result changes.

    What happens and why: Two rows and 200 units remain after repeated loads. Upsert means insert or update by a defined key. Idempotency means repeating an operation has the same effect as doing it once in this bounded case.

Takeaway: Retries are normal. Define business identity and write semantics so a repeated delivery does not become a new sale. Cleaning existing duplicates and preserving history need separate designs.

Where this is useful: Useful in reporting pipelines, payment-event handling and reliable data ingestion.

A fictional educational example, not a confirmed TWA assignment, assessment or learner outcome. Simplified to explore one idea; real situations need context and authorised procedures.

Optional deeper exploration: open the original technical workspace

Want to explore the technical model?

The guided story above introduces the idea. This original workspace lets you try actions in different orders. It has its own reset and does not inherit your story choices.

Append
Add each received row to the end of a destination.
Stable key
An identifier that continues to refer to the same business record.
Upsert
Insert a new key or update an existing key.
Idempotency
Repeating an operation produces the same final effect as doing it once.
TWA / The practice studio

Retry without multiplying records.

Learning objective: Use a stable key to make a pipeline rerun idempotent.

Explore · Observe · Explain

A fictional batch contains orders A=120 and B=80. Network failure may cause the same batch to run twice.

Evidence desk

Batch
A:120, B:80
Destination
Initially empty
Key
order_id identifies the same business record
Worked explanation & takeaway — available without JavaScript

Appending twice creates four rows and doubles the reported sum to 400. Upserting by order_id leaves two records and a sum of 200 after either run. Stable keys and defined update semantics matter; this toy model does not implement late arrivals, transactions or history.

Fictional educational model. No live system, personal assessment or confirmed TWA assignment. All actions run locally in your browser.

Explore the idea / pipeline

A pipeline should explain the second run

The two scenarios reveal reliability decisions; no database or live file is connected.

  1. Ingest

    Identify the source and record when it arrived.

  2. Transform

    Apply documented types and field rules.

  3. Check

    Test important expectations and isolate exceptions.

  4. Serve

    Publish an understood dataset with a status.

Read both scenarios and explanations

The same file arrives twice

Use the documented file/record identity and load rule to avoid silently duplicating business events. A replay should have a defined result.

An order has no identifier

Treat the missing key as an exception. Record the row and decision rather than inventing an identifier without a justified rule.

An educational illustration using fictional situations. It is not a confirmed TWA assignment, live system, assessment or promise of a particular tool.

Reliability is visible in the failure path

In the illustration below, a source file arrives twice. A carefully defined identifier and loading rule can prevent a replay from becoming an extra sale. A late or invalid row needs an explicit decision: reject it for review, quarantine it or process it under a documented rule. Quietly dropping it hides a business problem.

Data tests can assert expectations about missing values, uniqueness or relationships. dbt’s documentation provides examples of such assertions. A passing check only supports the expectation that was tested; it does not prove that every business assumption is right. Keep the check, result and response understandable to the person relying on the output.

Illustrative project brief

A replay-safe sample ingestion plan

Design a pipeline for a fictional daily order file. Document what should happen when the same file is received twice or a required identifier is missing.

What you could document

  • A source/target data model and field dictionary
  • A transformation outline in SQL, Python or pseudocode
  • Quality checks for identifiers and expected values
  • A rerun, failure-response and handover note

What makes the explanation useful?

A reviewer should understand why rerunning does not double-count records, where failed rows go and how a downstream user learns about a delay.

The learning plan determines actual activities and assessment. These deliverables are examples, not contractual curriculum outcomes.

Design for the next person who uses the data

A dataset handover should describe its fields, source, refresh expectation, quality checks and known limitations. It should also identify who responds when a run fails. An analyst should not have to guess whether a quiet dashboard means no activity or a stalled pipeline.

Deeper directions can include data operations, analytics engineering, warehouse design and distributed processing. Start with a small pipeline whose reruns and errors you can explain. Discuss your SQL/programming experience with TWA and compare this work with Data Analysis, where the emphasis is interpreting information rather than building its delivery path.

TWA learning direction / Confirm your starting point

Discuss a suitable starting point.

SQL and programming foundations help before a deeper pipeline plan. Bring a small data problem or explain any database experience. Ask about account access, compute costs, storage and the actual development environment.

Learners ready to progress from data or programming foundations into data systems.

Our physical centre is at No. 7 Nkpokiti Road, Ogui, Enugu, opposite Flamingo Mall. Learners elsewhere in Southeast Nigeria, across Nigeria and internationally can discuss suitable arrangements. Delivery, eligibility and availability need confirmation.

Ask about prerequisites and scope →

Existing TWA learning areas to discuss

  • SQL, relational databases and data modelling
  • Python for working with data
  • ETL / ELT and data pipeline concepts
  • Data quality, transformation and documentation
  • Cloud data storage and warehouse concepts

These are learning directions rather than a fixed syllabus. Confirm the actual tools, activities, equipment, schedule and assessment before committing.

Industry reading

References inform the explanations above. They do not imply a TWA partnership, accreditation or included software licence.

Questions about Data Engineering.

Do I need to be a data analyst first?

Analysis experience can help, but the key discussion is your SQL, programming and data-model understanding. TWA can discuss an appropriate foundation.

Does a scheduled job make a pipeline reliable?

Scheduling is only one part. Identity, checks, reruns, monitoring and an owner for failures also need clear decisions.

Which tools are guaranteed?

No particular orchestration platform or warehouse is promised here. Ask TWA to confirm the selected stack, access and costs.

Training options & enquiries

Understand your options.
Plan your next step.

Compare approved Nigerian packages or request a personalised private quotation for international training. Availability is confirmed with TWA.

Choose the training arrangement you want to discuss. TWA will confirm eligibility, learning needs and delivery arrangements with you.

Nigerian learners: approved fees

These rates apply only to Data Engineering. Prices for other programmes are confirmed by enquiry.

Approved Nigerian fees (NGN) for the training subjects listed below
Learning levelGroup TrainingPrivate Exclusive
Beginner₦300,000₦600,000
Intermediate₦700,000₦1,000,000
Advanced₦1,000,000₦1,500,000

Group Training includes

  • Live training sessions
  • Session recordings
  • Learning materials

Private Exclusive includes

  • Live training sessions
  • Session recordings
  • Learning materials
  • Projects
  • Interview preparation
  • CV preparation

Before you enquire

  • Group Training requires more than 20 participants. Group formation, schedules and places need confirmation.
  • Standard training lasts a maximum of 2 months, except Private Exclusive training. Confirm the dates for your selected plan.
  • Private training may involve online and physical sessions in a hybrid format, subject to availability and confirmation. Private Exclusive duration and schedule are agreed with TWA.
  • Enquiry required. A listed programme or price does not confirm an active cohort or an available training slot.
Prepare a WhatsApp enquiry directly — all approved options

Each link includes the programme, level, package and fee. Review the draft in WhatsApp and press Send there.

Data Engineering — choose level & package

Select an option to prepare a WhatsApp enquiry with the programme, level, package and approved fee.

International learners / Private training

A private learning plan.
A personalised quotation.

Discuss private training around your current experience, learning objectives, desired specialisation, preferred schedule and time zone, duration, delivery arrangements and individual learning needs. TWA will confirm what can be offered before you commit.

International training uses personalised quotations. Nigerian fees are not applied or converted into another currency. Schedules, duration and delivery must be agreed with TWA; a request does not confirm availability.