Skills and Tools
Every skill you need for a data career in South Africa, filtered by role. Click a tab to see what applies to you.
Foundation Skills
Required across all four data roles.
Python
The primary language for data work. Every data role requires Python at some level. Data Scientists and AI Engineers live in it; Engineers use it for pipelines; Analysts use it for automation and advanced analysis.
- Data types, loops, functions, list comprehensions
- pandas: read, filter, group, merge, reshape DataFrames
- numpy: arrays, broadcasting, vectorised operations
- Write clean reusable functions and modules
- Virtual environments (venv / conda)
SQL
SQL is the language of data. Every data role requires it. Analysts query and aggregate. Engineers design and optimise. Scientists and AI engineers use it to pull and prepare training data.
- SELECT, WHERE, GROUP BY, ORDER BY, HAVING
- JOINs: INNER, LEFT, RIGHT, FULL OUTER
- Window functions: ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, SUM OVER, AVG OVER
- CTEs (WITH clauses) and subqueries
- Query optimisation: EXPLAIN ANALYZE, indexes, partitioning
- Date arithmetic and string functions
Git and Version Control
Version control is a professional requirement for every data role. Git tracks changes, enables collaboration and is the foundation of all CI/CD pipelines.
- git init, add, commit, push, pull
- Branching: git branch, checkout, merge, rebase
- Pull requests and code review workflow
- Resolving merge conflicts
- .gitignore best practices for notebooks and data files
Statistics and Probability
The mathematical foundation under all data roles. Without statistics you can describe data but not understand it. Required to design experiments, evaluate models and interpret results correctly.
- Descriptive stats: mean, median, std, percentiles, skewness
- Probability distributions: normal, binomial, Poisson, uniform
- Central Limit Theorem and sampling
- Hypothesis testing: z-test, t-test, chi-squared, ANOVA
- Confidence intervals and p-values
- Correlation vs causation
- Bayes theorem and conditional probability
Data Analyst Skills
Tools and techniques for analysis, reporting and visualisation.
Excel and Google Sheets
Still the most widely used data tool in SA businesses. Every analyst must be proficient. Used for quick analysis, reporting, and communicating results to non-technical stakeholders.
- Pivot tables and pivot charts
- VLOOKUP, INDEX/MATCH, XLOOKUP
- Power Query: import, clean and reshape data
- Conditional formatting and data validation
- Array formulas and dynamic arrays (FILTER, UNIQUE, SORT)
- Dashboard building with charts and slicers
Power BI
The dominant BI tool in South African corporates. Most data analyst job ads in SA list Power BI as required. Learn this before Tableau if you are targeting the SA market.
- Connect to data sources: Excel, SQL, SharePoint, APIs
- Data modelling: star schema, relationships, calculated columns
- DAX: CALCULATE, SUMX, FILTER, RELATED, time intelligence
- Report and dashboard design: slicers, drill-through, bookmarks
- Row-level security (RLS)
- Publishing and sharing via Power BI Service
Tableau
Leading global BI tool with strong presence in consulting and international companies. Valued in SA at firms like Deloitte, PwC and multinational corporates.
- Connecting to data sources and live connections
- Calculated fields and LOD expressions: FIXED, INCLUDE, EXCLUDE
- Dashboard actions and parameter controls
- Tableau Prep: cleaning and reshaping data
- Publishing to Tableau Public for your portfolio
Data Storytelling
The ability to translate data findings into clear, actionable business decisions. The most underdeveloped skill among junior data professionals. Technical accuracy means nothing if the business cannot act on it.
- Structure a data narrative: context, finding, implication, recommendation
- Choose the right chart for the message (not the data)
- Design principles: colour, hierarchy, whitespace, annotation
- Tailor communication for technical vs non-technical audiences
- Executive summary: one page, one insight, one action
A/B Testing and Experimentation
Rigorous experiment design is how data teams prove causation, not just correlation. Every SA bank, insurer and large retailer runs experiments. This skill moves you from reporter to decision-maker.
- Experimental design: control vs treatment, randomisation
- Sample size and statistical power calculations
- Running the test: z-test, t-test, chi-squared by context
- Interpreting results: significance, effect size, practical significance
- Multi-variate testing (MVT) basics
- Common pitfalls: p-hacking, peeking, survivorship bias
Google Analytics 4
Standard tool for web and product analytics. Required if you are targeting digital, e-commerce or media companies in SA.
- GA4 event-based data model vs Universal Analytics
- Setting up events, conversions and funnels
- Audiences, segments and user properties
- GA4 + BigQuery export for raw event data analysis
- Looker Studio dashboards on top of GA4
Data Engineer Skills
Building and maintaining data pipelines, warehouses and infrastructure.
Data Modelling
How data is structured for analytical workloads. Good data modelling makes queries fast and dashboards reliable. Bad modelling creates technical debt that breaks at scale.
- Star schema: fact and dimension tables
- Slowly changing dimensions (SCD Type 1 vs Type 2)
- Normalisation vs denormalisation trade-offs
- dbt: models, tests, documentation, incremental builds
- OLTP vs OLAP design differences
Data Warehousing
Cloud data warehouses are where analytical data lives at scale. Every major SA company is migrating off on-premise to cloud. BigQuery, Snowflake and Redshift dominate the market.
- Partitioning and clustering for query performance
- Cost optimisation: slot usage in BigQuery, credits in Snowflake
- Loading data: batch vs streaming ingestion patterns
- Schema management: schema-on-read vs schema-on-write
- Data vault and OBT (one big table) approaches
Pipeline Orchestration (Airflow)
Airflow is the industry standard for scheduling and orchestrating data pipelines. A reliable pipeline is idempotent, observable, and fails gracefully.
- DAGs: tasks, dependencies, operators
- Scheduling with cron expressions
- Sensors, triggers and XCom for inter-task communication
- Error handling: retries, SLAs, alerting
- Idempotency: what it means and why it matters
- Running Airflow locally with Docker Compose
Apache Spark
The engine for large-scale data processing. When pandas runs out of memory, you need Spark. Used for batch processing of millions to billions of rows.
- Spark architecture: driver, executors, partitions
- RDDs vs DataFrames vs Datasets
- Transformations (lazy) vs Actions (eager)
- Joins in Spark: broadcast, sort-merge, shuffle
- Reading/writing Parquet, Delta, CSV at scale
- Spark on Databricks (most common SA deployment)
Apache Kafka and Streaming
Real-time data streaming for event-driven architectures. Used by SA banks and telecoms for fraud detection, transaction monitoring and real-time dashboards.
- Kafka concepts: topics, partitions, consumer groups, offsets
- Producers and consumers in Python
- Exactly-once vs at-least-once delivery semantics
- Stream processing: windowing, aggregations
- Kafka Connect for source and sink connectors
Cloud Platforms
Production data infrastructure lives on cloud. Learn one platform deeply. GCP is common at SA fintechs and startups; Azure is dominant in SA enterprise; AWS has strong international presence.
- Storage: object storage (GCS/S3), managed databases
- Compute: VMs, serverless functions, containers
- Managed services: Cloud Run, Lambda, Azure Functions
- IAM: roles, permissions, service accounts
- Infrastructure as Code: Terraform basics
- Cost monitoring and budget alerts
Docker and Containers
Containers are how data applications are packaged and deployed consistently. A pipeline that works on your laptop should work the same in production. Docker makes this possible.
- Docker concepts: images, containers, volumes, networks
- Writing a Dockerfile for a Python data app
- docker-compose for multi-service local development
- Container registries: Docker Hub, GCR, ECR
- Kubernetes: pods, deployments, services (conceptual level)
dbt (Data Build Tool)
dbt has become the standard for data transformation in the modern data stack. It brings software engineering practices (testing, documentation, version control) to SQL-based transformation.
- Models: ref(), sources, materialisation (table, view, incremental)
- Jinja templating for dynamic SQL
- dbt tests: not_null, unique, accepted_values, relationships
- Documentation: schema.yml, descriptions, lineage graph
- dbt Cloud: scheduling, CI/CD, IDE
NoSQL Databases
Not all data fits in relational tables. NoSQL databases handle document, key-value, graph and time-series data at scale. Used heavily in SA at fintechs and telecoms.
- Document stores: MongoDB queries, indexes, aggregation pipeline
- Key-value stores: Redis data structures and use cases (caching, sessions)
- When to use NoSQL vs relational: access patterns, consistency requirements
- CAP theorem: consistency, availability, partition tolerance
- Time-series databases for IoT and event data
APIs and Data Ingestion
Most data does not come from databases. It comes from REST APIs, webhooks and third-party services. Engineers build the connectors; analysts use them for enrichment.
- REST API fundamentals: GET, POST, PUT, DELETE, headers, auth
- Pagination, rate limiting and retry logic
- Authentication: API keys, OAuth 2.0, JWT tokens
- Parsing and normalising nested JSON
- Building a connector with Airbyte or Singer
- Writing data to Parquet for efficient downstream processing
Data Scientist Skills
Machine learning, statistical modelling and experimentation.
Feature Engineering
The skill that most separates good data scientists from great ones. Raw data almost never has the signal the model needs. Feature engineering creates it.
- Numerical features: binning, log transforms, normalisation, standardisation
- Categorical features: one-hot, ordinal, target, frequency encoding
- Date/time features: day of week, month, lag features, rolling averages
- Interaction features: ratios, products, differences
- scikit-learn Pipeline: column transformers, custom transformers
- Feature selection: RFE, LASSO, mutual information, variance threshold
MLOps
Getting a model to 80% accuracy is 20% of the work. Deploying it reliably, monitoring it and retraining it when performance degrades is the other 80%.
- Experiment tracking: logging parameters, metrics, artifacts in MLflow
- Model registry: versioning, staging, production promotion
- Model serving: FastAPI endpoint, REST API, batch scoring
- Feature stores: Feast, Tecton (conceptual)
- Model monitoring: data drift (PSI), concept drift, performance degradation
- Retraining triggers and CI/CD for ML pipelines
R
R is widely used in SA for statistical analysis, actuarial work and academia. Valued at Discovery, insurance companies, academic research roles and the NRF. Python-first, but R as a second language is a competitive advantage.
- tidyverse: data manipulation with dplyr and tidyr
- ggplot2: publication-quality visualisations with a grammar of graphics
- Statistical modelling: lm(), glm(), survival models
- R Markdown for reproducible analysis reports
- Shiny: interactive web applications from R
NLP and Text Analytics
Natural language processing turns unstructured text into structured insights. Used in SA for customer feedback analysis, document processing, chatbots and compliance monitoring.
- Text preprocessing: tokenisation, stopword removal, lemmatisation, stemming
- Vectorisation: Bag of Words, TF-IDF, word embeddings (Word2Vec, GloVe)
- Sentiment analysis: lexicon-based and ML-based
- Text classification with scikit-learn and fine-tuned transformers
- Named entity recognition (NER) with spaCy
- Semantic search with sentence transformers
AI / ML Engineer Skills
Deep learning, LLMs, model deployment and ML systems design.
Machine Learning
The core technical skill of a data scientist. Understanding why algorithms work, when they break, and how to evaluate them properly separates a data scientist from someone who copies scikit-learn examples.
- Supervised learning: regression (linear, ridge, lasso) and classification (logistic, SVM, decision trees)
- Unsupervised learning: k-means, DBSCAN, PCA, t-SNE
- Ensemble methods: random forests, gradient boosting, stacking
- Cross-validation: k-fold, stratified k-fold, time-series split
- Imbalanced datasets: SMOTE, class_weight, threshold tuning, PR-AUC vs ROC-AUC
- Hyperparameter tuning: GridSearchCV, RandomizedSearchCV, Optuna
- Model explainability: SHAP values, feature importance, LIME
Deep Learning
Neural networks and deep learning power modern AI. Essential for AI/ML Engineers and increasingly expected at senior DS level. PyTorch is preferred in research; TensorFlow/Keras in industry.
- Feedforward networks: layers, activations, loss functions, optimisers
- Backpropagation and gradient descent from scratch in numpy
- CNNs: convolutional layers, pooling, transfer learning (ResNet, EfficientNet)
- RNNs and LSTMs: sequence modelling, vanishing gradients
- Transformers: attention mechanism, self-attention, positional encoding
- Batch normalisation, dropout, learning rate scheduling
- Experiment tracking with Weights and Biases
LLMs and Generative AI
Large language models are now a core AI engineering skill. The ability to build production applications on top of LLMs via APIs, RAG and fine-tuning is highly valued in SA and globally.
- Prompt engineering: zero-shot, few-shot, chain-of-thought
- RAG (Retrieval Augmented Generation): chunking, embeddings, vector search, reranking
- Fine-tuning: LoRA, QLoRA on open-source models (Mistral, LLaMA)
- Vector databases: Pinecone, Chroma, Weaviate, pgvector
- LLM evaluation: RAGAS, hallucination detection, faithfulness metrics
- Agents and tool use: function calling, ReAct pattern
Model Deployment and Serving
Building a model is step one. Serving it reliably at scale to thousands of concurrent users is where AI engineering begins. Latency, throughput, fallback strategies and observability all matter.
- REST API model endpoints with FastAPI + pydantic validation
- Batching: dynamic batching for throughput, synchronous vs async
- GPU serving: Triton Inference Server, model quantisation (INT8, FP16)
- A/B testing between model versions (canary deployments)
- Caching: feature caches, prediction caches for repeated inputs
- SLA monitoring: p50, p95, p99 latency, error rate, throughput
Vector Databases and Embeddings
The infrastructure layer powering semantic search, RAG systems and recommendation engines. Understanding embeddings and vector similarity search is now a required AI engineering skill.
- What embeddings are and why they capture semantic meaning
- Embedding models: sentence-transformers, OpenAI, Cohere, BGE
- Similarity metrics: cosine similarity, dot product, L2 distance
- ANN search: HNSW, IVF, PQ (approximate nearest neighbour)
- Chunking strategies for documents: fixed, semantic, hierarchical
- Hybrid search: combining vector similarity with keyword (BM25)
ML Systems Design
AI/ML Engineers are expected to design end-to-end ML systems at scale. This is the system design interview for ML roles: feature pipelines, model training, serving, monitoring and feedback loops.
- Feature pipeline: real-time vs batch feature computation, point-in-time correctness
- Training pipeline: data versioning, experiment tracking, distributed training
- Serving: online (low latency) vs offline (batch scoring) patterns
- Feedback loop: labelling pipelines, online learning, champion/challenger
- Monitoring: feature drift (PSI), prediction drift, model performance degradation
- Cost vs accuracy trade-offs: model compression, distillation, quantisation